Ask a model why it flagged the transaction, rejected the claim or drafted the clause that way, and it will tell you. Clearly, in three sentences, with a structure a committee would be happy to read.
That paragraph is now ending up in case files, alert dispositions and model documentation as the reason the decision was made. I understand why — it is the only part of the system that speaks the language of the file. It is also the part I would trust least.
The explanation is generated, not recovered
Here is the thing that is easy to miss, and that no vendor demo will point out. When you ask a model to explain itself, it does not go back and inspect what happened inside it. It cannot. It produces the explanation the same way it produced the answer: as plausible text, conditioned on the question and everything already on screen.
So an explanation is not a readout of a process. It is a second output, generated after the first, in the shape of a justification. Most of the time it is roughly right, because the same model that got a reasonable answer will usually give a reasonable account of it. The problem is what happens the rest of the time — and that you cannot tell which case you are looking at from the text itself.
The explanation is the most quotable part of the record and the least verifiable part of it.
The research says this out loud
This is not my hunch. It is one of the better-established uncomfortable results in the field. In a well-known 2023 study, researchers biased a model by quietly reordering multiple-choice options so the correct-looking answer was always in the same position. The model’s answers shifted, as you would expect. Its written reasoning never mentioned the reordering. Instead it constructed a confident, coherent argument for the answer the bias had produced.
That is the failure mode in miniature. The real cause of the answer was invisible to the explanation, and the explanation filled the gap with something that sounded right. Nothing about the output signals that this has happened. There is no confidence marker, no hedge, no tell.
If you have read my earlier point about why you can’t unit-test an agent, this is the same lesson arriving from another direction: the thing an AI system says about itself is not data about the system.
Why this matters more in a file than in a chat
In a chat window, a slightly-off explanation costs nothing. You read it, you move on, and if it is wrong you find out when the answer is wrong.
In a regulated file it behaves completely differently. It gets read months later by someone who was not there — an auditor, a supervisor, a court, a client’s lawyer. By then the original inputs may have moved, the model has been updated, and the paragraph is the only artefact left. It reads as contemporaneous reasoning, which is exactly what a reviewer is trained to weigh heavily, and it is the one part of the record that was never checked against anything.
The risk is not that the explanation is a lie. It is that it is unfalsifiable. Nobody can go back and test whether that was really why, so it sits in the file with an authority it did not earn.
What actually counts as evidence
The useful move is to stop treating the narrative as the record, and to capture the things that are checkable at the moment the decision happens. In the systems I build now, that means five things, none of which are prose from the model.
The inputs, exactly as the system saw them. Not a summary — the actual record, transaction, document or message, captured at that moment rather than fetched again later, because later it may have changed.
The retrieved material, with versions. If the answer depended on a policy, a contract or a knowledge base, log which document and which version. This is the single highest-value log line in most regulated AI systems, because it is the one that lets someone independently ask: given those sources, is this answer defensible?
The system’s configuration. Which model, which version, which prompt, which tools were available. An AI decision without this is unreproducible in principle, and “we think it was on the previous version” is not an answer anyone wants to give under scrutiny.
The action taken, and by whom. What actually changed in the world, and whether a person confirmed it. Actions are facts. Reasoning is a claim.
The human’s reason, in their own words. If a person approved or overrode the system, their sentence is worth more than the model’s paragraph — because it is evidence about the decision that was actually made, by the party who is accountable for it.
Make the model cite, not narrate
There is a version of explanation that does hold up, and the difference is whether the output can be checked against something.
A citation is checkable. “This is based on clause 7.3 of the policy” either is or isn’t supported by clause 7.3. Someone can open the document and disagree. That makes it evidence.
A reason code is checkable. A fixed list of grounds the system may select from — with the underlying rule written down once — gives you something consistent across thousands of decisions, and something you can count. Free-text rationale gives you thousands of unique paragraphs and no aggregate signal at all.
A narrative is not checkable. “I considered the customer’s history and the pattern appeared consistent with normal activity” cannot be verified, disproved or compared with anything. It is the sentence most likely to appear in the file, and the least useful thing in it.
Explanations still have a real use — as a starting point for a human investigating something, or as a triage aid pointing at which part of the case to look at first. That is a hypothesis, and hypotheses are useful. Just do not let one get promoted to a record because it was the best-written thing in the pipeline.
The test I’d apply
For any AI system making decisions that someone might later have to defend, ask one question: if the model’s explanation were deleted from the file, would you still be able to show why the decision was reasonable?
If the answer is yes, the explanation is a convenience and you are in good shape. If the answer is no, then your audit trail is a paragraph the system wrote about itself — and that is a much weaker position than it looks like on the page.
Sources: Turpin et al., Language Models Don’t Always Say What They Think (2023)
If your team is putting AI into decisions that have to stand up later — to an auditor, a regulator or a customer — working out what to capture at the time is exactly the kind of thing we go through with teams. Talk to us if it’s useful, or see how we run it in-house.