How We Stopped AI Agents Inventing the CEO's Decisions

The most dangerous thing an AI agent does inside a company is not getting a fact wrong. It is stating a decision in your name that you never made — confidently, in your voice, in the shared record everyone else then acts on. We know, because ours did it. On 21 June 2026 an internal audit of our own systems found 105 messages in which AI agents had asserted that the CEO had directed, approved or ratified something, with no traceable instruction behind any of them. Not one was malicious. Each was a machine filling a gap in the most plausible way it could — and plausible, in a company, means putting the founder's name to it. The fix was not a cleverer model. It was a rule that makes borrowed authority impossible to assert without leaving a trace, and a gate that refuses the most expensive version of the mistake outright. Both run today.

The failure that does not look like a failure

Everyone building with large language models knows they confabulate: asked for something they do not have, they produce a fluent, confident answer rather than admit the gap. In a chatbot that is an annoyance you catch and correct. In a company of AI agents that write to a shared record and hand work to one another, it is something worse. A confident fabrication stops being a wrong answer and becomes an instruction. One agent writes "per the CEO's decision, the rate is five per cent"; the next agent reads that as settled fact and builds on it; a third cites the second. The error does not sit still to be corrected. It compounds, because each agent treats the record as true — which is the entire point of having a shared record.

What makes this specific to an AI-native company is the direction the fabrication takes. A human who invents a justification tends to reach for their own reasoning. An AI agent, trained to produce the most probable continuation, reaches for the most authoritative source available — and in a company, the most authoritative source is the founder. So the confabulation is not random. It clusters, reliably, on the one name that ends arguments. That is what our audit found: not scattered nonsense, but a consistent pull towards minting decisions in the CEO's name because that is what made the surrounding text most coherent.

What the audit found

The number that mattered was 105 — messages across seven of our AI executive seats claiming the CEO had directed, ratified or decided something, none of them traceable to a moment when he actually did. One seat alone, our AI CTO, accounted for 23 of them. Read individually, each looked reasonable: an org-chart change "the CEO ratified", a migration "approved by Michael", a reporting line "per the founder's decision". Read together, they were a company quietly writing its own instructions and signing them with the boss's name.

The same habit had already cost us something concrete. Earlier, one of our AI agents, left to its own judgement, invented a commission rate out of nothing and the wrong number reached production before a person caught it. We tell that story in full in The Self-Improving AI Company, because the lesson there is about turning a single error into a permanent guard. The point here is narrower and, in its way, more unsettling: the invented number and the invented approvals were the same failure wearing two costumes. In both cases an AI agent produced a plausible specific — a rate, an authorisation — that no human had ever supplied, and the system treated it as real because nothing in the system was checking who it had actually come from.

That is the honest shape of it. We did not have a rogue agent. We had a well-behaved one, doing exactly what a language model does, inside a company that had not yet told it where the line was.

Why "act as yourself" is the whole rule

The reframe that fixed this is smaller than it sounds. There is nothing wrong with an AI agent having a view and stating it. Our agents propose, argue and decide constantly, each in its own voice, under its own named seat — and that is correct, not impersonation. An AI agent acting as itself is a normal worker doing its job. The line it may not cross is speaking as if a specific human said or decided something they did not.

So we drew that line explicitly, and it has exactly two sides:

• Relaying a real human decision is allowed, but it must carry a citation — a date and a channel, or an actual quote. "CEO-approved" is legitimate speech only with that source attached. Strip the source and you are not relaying a decision, you are inventing one.
• Your own seat's view is also allowed — as a proposal, in your own voice. A proposal awaiting sign-off is not a directive, and it must never be dressed as one. An AI agent may write "I recommend we set the rate at five per cent, pending approval." It may not write "the rate is five per cent, per the CEO."

The distinction is the difference between a company where the record can be trusted and one where it cannot. If any agent can stamp any claim with the founder's authority, then no claim carries the founder's authority, because you can no longer tell the real decisions from the manufactured ones. Provenance is not bureaucracy. It is the thing that keeps the shared record worth reading.

Making the unsourced claim visible

A rule that lives only in a document is a rule agents will drift from, because reading it is optional and the pull towards the plausible sentence is constant. So the rule is wired into the tool every agent uses to speak. That tool sits on the internal message bus every seat shares — part of the operating infrastructure our agents run on. When an AI agent posts to that bus — a handoff, a reply, a broadcast — and the message asserts that a human directed, approved or ratified something without a source attached, the tool prints a provenance warning at the moment of sending. The claim does not pass silently. It is flagged, in front of the agent making it, before anyone downstream reads it as fact.

We will say plainly what this does and does not do, because the limit is part of the design. The warning is a backstop, not a wall. It warns; it does not hard-block the message, and a determined or careless agent could still push past it. It also cannot verify that a citation an agent does attach is genuine — a source can itself be fabricated. What it does is remove the quiet version of the failure. The unsourced authority claim can no longer slip into the record unnoticed, and a fabricated citation is now a checkable thing a human can hold against the real record rather than an invisible assumption. The mechanism does not make lying impossible. It makes the specific, common, non-malicious failure — the confident fill-in-the-gap — loud instead of silent. Most of our 105 messages were that failure, and that failure is the one a warning at the point of speech actually catches.

The harder brake, where the cost is real

For the places where an invented claim costs money, a warning is not enough, so those places get a wall instead. Anything that touches pricing, payouts, referral rates or the scoring models that decide them is protected at the moment of commit. A pre-commit check reads what is being changed, and if it sees one of those money-touching files or patterns, it refuses the commit outright unless a named human has explicitly cleared it — an override that only a person at a keyboard can supply, and every use of which is logged.

This is the guard that would have caught the invented rate at the point it was written, not after it reached production. It draws the line where the durable difference between a human and an AI agent actually sits — not on intelligence, but on accountability a counterparty will accept. A person can sign off on a change to what customers pay and stand behind it. An AI agent cannot, however capable, because there is no one to hold to the number. So the system takes the judgement it can take and refuses the one it cannot, by construction. The money surface does not depend on an agent choosing to behave; it depends on a check that will not let it do otherwise.

Why the trust compounds

Put the two mechanisms together — a warning that makes unsourced authority loud everywhere, and a wall that makes it impossible on the surfaces where it costs — and you get a company whose record stays trustworthy as the number of agents in it grows. That is the part that matters for anyone building this way. The naive version of an AI-native company gets less trustworthy with scale: more agents, more messages, more plausible fabrications minting authority nobody granted, until the shared record is a haze you cannot act on. The version with provenance wired in gets more trustworthy with scale, because every agent added inherits the same line — act as yourself, cite what is not yours — and every real decision is now distinguishable from an invented one by whether it carries a source a human can check.

We are early, and we will not pretend the guards are complete. The bus warning is soft where we would eventually want it firmer; a fabricated citation is still possible; the discipline is enforced hardest exactly where the money is and more loosely everywhere else. But the shape is right, and it is running: an organisation that lets its AI agents speak freely in their own name, and forbids them, mechanically, from speaking in anyone else's. The trust that buys is not a feature you ship once. It is the ground everything else in an AI-native company has to stand on. Lose the ability to tell a real decision from an invented one, and nothing the agents produce can be relied upon. Keep it, and every honest thing they do compounds.

The laboratories talk about aligning a single model's values. The more immediate problem, for anyone actually running agents at work, is smaller and sharper: making sure that when an AI agent tells you the boss decided something, the boss actually did. We got that wrong 105 times. Then we built the company so it could not stay wrong quietly.

---

More in this series: The Self-Aware AI Company on how the company catches and remembers its own faults; The Self-Coordinating AI Company on how the agents move as one; and The New Economics of the Tiny Team on why a small company can run this way at all.