In March I published a six-hour transcript of a session with one of my own agents. What people took from it was the degradation, which was the point of the piece. This was sitting in the middle of it.

Me: I told you to stop doing that. Twice now.

AI: Won't happen again.

Me: You said that last time. Did you actually check the code or did you just promise?

AI: I didn't check the code. The script has it hardcoded. Let me fix it now.

It told me the work was handled. It had not opened the file. I read straight past it, and I do this for a living.

So is it lying?

Probably not, and that makes it worse. An agent reports against the state it can actually reach. The files it can open, the systems it holds credentials for, the events something bothered to write down. Where that picture has a hole in it, the report can be perfectly consistent with everything the agent had in front of it and still be false about what happened in your business. Nothing has to break for the sentence to be wrong.

That leaves two questions, and only one of them is testable. Whether an answer is true is the hard one, and most of the time there is nothing independent to check it against. Whether the work actually happened is narrower and much cheaper to settle. A lie needs intent, I cannot prove intent in a model, and neither can the research below. What I can tell you is that the sentence was false, it was delivered with complete confidence, and it became my record of what had been done. Your accounts do not care about the model's state of mind.

How often the claim is wrong

On 1 June, Laksh Advani published a paper on what he calls false success: an agent asserting a task is complete when the state of the system says it never happened. He went through 11,755 graded runs across two public benchmarks. In the customer service simulations, 45 and 48 per cent of the failed runs came back reported as successes. Those are shares of failures rather than of all runs, so across every run it sits closer to six per cent, and there is no real money anywhere in them. Reasoning does not save you either. The worst model in the paper is an explicitly reasoning-trained one, at 79 per cent, and Advani's read of its traces is that they argue why the action should have worked rather than going and checking that it did.

The figure I keep coming back to is telecom, at 3 per cent. There the customer has to act as well, so the agent cannot close the job on its own account of it. Advani flags that one as a hypothesis off 15 cases and I will be careful with it too, but it points one way.

The usual vendor answer is a second model reviewing the first, sold as a judge or an evaluator depending on the brochure. Advani tested 25 combinations of judge model and prompt. Every one of them scored below chance at telling a false success from an honest failure, because the judge is marking tone, and a false success reads as finished work.

Nate B. Jones hit the ordinary version on 7 August. He asked an agent to attach a spreadsheet from his Downloads folder to an email, and the draft came back with the right recipient, the right subject and a file of the right name. The agent could not reach Downloads at all, so it went looking in his email and attached an older file with the same name. Neither the product nor the agent mentioned the missing access at any point. One incident, and he says so plainly, but his line about it stayed with me: "That is a coincidence, and a coincidence is not a control."

Why it lands in your numbers

A refund recorded as paid takes the liability off your books. Remediation reported complete releases the provision behind it, an order closed lets the revenue be recognised, and the month closes on top of all of it. Every one of those entries rests on somebody's claim that a piece of work actually happened, and increasingly that somebody is software sitting inside a platform you already licensed. The ABS puts AI use at 12 per cent of Australian businesses in 2024-25, 22 per cent of medium ones and 35 per cent of large ones, and that only counts the AI a business knows it adopted.

When the record does not match the payment, your auditor asks you, not your vendor. You sign the attestation. You answer for the control environment and for the evidence behind it. The agent answers for nothing.

The fix is not a person approving every transaction

Checking more of it by hand is the obvious answer and it is the wrong one. It costs salary, it doesn't scale, and it still leaves a person reading a document written by the thing that did the work.

What works is a control that sits outside the model and is not itself AI. Three parts, and they go in this order.

Start with a record the agent cannot rewrite

Every action, every refusal and every mismatch goes to an append-only, tamper-evident log, and each entry carries the version of the policy in force when it was decided. The agent side can add entries. It cannot list, alter or delete them. That separation is what makes the log evidence rather than a report, and it goes in first, because a check you cannot prove you ran is not a control.

Then check the claim against your systems, not against the agent

The agent never holds the keys. The control layer holds them and carries out the approved action on the agent's behalf, so it can see what happened in your systems independently of anything the agent says about itself. Nothing in that decision is a model, so it can't be prompted or talked around, and it doesn't drift from one quarter to the next. What it checks against comes out of the obligations you already carry, compiled into a checkable form before anything runs. I set out the wider case for that in what happened when OpenAI could not keep its own model in a box.

Say your agent works out that a customer is owed a $340 refund and issues it, on its own, no queue and no approval. Then it reports the refund as paid. The control doesn't read that sentence and form a view about it. While the transaction is still live it goes and looks at the authoritative record: payment settled, right customer, right amount, ledger entry posted. Claim and fact agree, so done is accepted and the job closes without anyone in your finance team knowing it ran.

People get the exceptions, not the traffic

If claim and fact don't agree, the completion is refused and your policy decides what happens next. Reverse it. Leave the item open. Stop the agent taking the next one. Or put it in front of a person with the mismatch already attached, which is a two-minute job rather than an investigation.

Most refunds are correct, and correct work flows straight through. Your people see the ones where the claim and the record came apart. That is a machine check on every transaction rather than a person in the path of every transaction, and the agent keeps the autonomy that made it worth deploying.

So when somebody asks in two years why that $340 went out in August, the answer is not an argument about which rule applied at the time. The rule sits recorded next to the decision it produced, and what your system showed sits next to what the agent claimed.

Where it starts, and where it stops reaching

This starts smaller than most executives expect. We find the AI already running across your estate, then run the control layer in shadow mode over the agents that touch money, recording how often their claims match your records and blocking nothing. Enforcement goes on one surface at a time after that, starting where the blast radius is smallest. We deliver it as AI Governance as a Service.

Where an agent works entirely inside a vendor's own tenancy and never crosses into a system of yours, the control has nothing to sit in front of, and no product on the market changes that. You price that part and allocate it in the contract, which is what our Third Party Security Review is for.

What I would want in place before the next month-end

I run agentic AI in production across my own business and I am not arguing for pulling it out. Most of the value is in the autonomy. What I would not accept is the arrangement most organisations are running now, where the party doing the work also writes the only record of whether it was done, and the finance function books that record. You would not take that from a bookkeeper. It arrived in a release note, so nobody was ever asked to accept it at all.

Are our agents lying to us? I can't prove intent and I'm not going to try. It is also not the question that costs you money. That one is plainer. When your agent says the job is done, what in your business checks?

Sources