# You can’t unit-test an agent. So how do you know it works? > Normal software fails loudly and repeatably. Agents fail quietly, differently each time, and only on the cases you didn’t imagine. Here’s what testing one actually looks like. By Arthiq — undefined · AI at work Source: https://arthiq.co/blog/you-cant-unit-test-an-agent --- Every engineer I know has the same reflex when they ship something: write the tests. Same input, same expected output, green tick, move on. It’s a good reflex, and it’s the first thing that breaks when you start building with agents. Run the same task through an agent twice and you may get two different routes to the same place — or the same route to two different places. There is no single correct output to assert against. Which leaves a genuinely uncomfortable question that I think most teams are quietly avoiding: how do you know the thing works? ## Why the old test doesn’t fit A unit test makes three assumptions. That the output is deterministic. That correctness is exact. That the failure will show up as an error. An agent breaks all three. It’s non-deterministic by design, so the same prompt can take a different path on Tuesday. Correctness is usually a judgement — the summary is good enough, the email is appropriately worded, the right record got updated — not a string match. And worst of all, the failures are silent. An agent that misunderstands a task doesn’t throw an exception. It confidently does the wrong thing and reports success. > Conventional software tells you when it’s broken. An agent tells you it’s finished. That last one is the part that catches people out. Your monitoring is green, your error rate is zero, and the agent has been quietly filing things in the wrong place for a fortnight. ## What an eval set actually is The replacement for the unit test is the eval set, and it’s much less exotic than it sounds. It’s a collection of real tasks with an agreed view of what a good outcome looks like — thirty to a hundred of them is plenty to start. **Build it from real work, not imagined work.** The single most useful thing you can do is take actual requests from the last month — real emails, real tickets, real messy inputs — rather than the tidy examples you invented while building. Invented cases test the agent against your assumptions. Real ones test it against your business. **Include the cases you’re afraid of.** Half the value of an eval set is the awkward end: the ambiguous request, the one with missing information, the one where the right answer is to stop and ask. If your set only contains tasks the agent handles well, all it measures is your optimism. **Grade the outcome, not the wording.** For most tasks you don’t care what the agent said, you care what changed — was the right record updated, was the refund the right amount, did it correctly decline to act. Where the output really is prose, a rubric works better than an exact match: was it accurate, did it stay in scope, did it avoid inventing facts. Some of that grading can be automated, including by another model; some of it needs a person. Both are fine. Neither is optional. The point of the set isn’t to prove the agent is perfect. It’s to give you a number that moves when you change something, so you can tell an improvement from a hunch. ## The trace is the real artefact When an agent gets something wrong, the output tells you almost nothing. The trace tells you everything — what it was asked, what it retrieved, which tools it called with which arguments, what came back, and what it decided next. I now treat the trace as a first-class part of the system rather than debug output. If a task went wrong, I want to see the exact step where the reasoning turned, because the fix is almost always different depending on where that was. Bad retrieval is a data problem. A tool called with the wrong arguments is an interface problem. A sensible-looking chain that arrives somewhere stupid is a scoping problem — and, as I’ve [written before](/blog/the-leash-not-the-model), usually a boundary you didn’t draw. Without traces you end up doing the worst kind of engineering: changing the prompt, running it again, and hoping. ## Failures you only see in production Even a good eval set is a rehearsal. Some things only appear once real people are involved. **Inputs you never imagined.** Users phrase things in ways no one on the team would. They paste in half a thread, they ask two questions at once, they use internal shorthand. **Drift.** Your data changes, your processes change, and a model provider updates something underneath you. An agent that was right in March can be subtly wrong in July with no code change at all. **The compounding path.** In a multi-step task, a small early misreading doesn’t stay small. It gets carried forward and confidently built upon. So the eval set needs to be a living thing. Every real failure becomes a new case in it. That’s the loop that actually makes an agent better over time — not a cleverer prompt, but a growing record of the ways it has genuinely gone wrong. ## What to measure before you ship You don’t need a dashboard of twenty metrics. Four questions are usually enough. **How often does it get the task right?** Your eval set score, tracked over time rather than checked once. **How often does it do something it shouldn’t?** Out-of-scope actions matter far more than imperfect ones, and should be counted separately. **How often does it correctly stop?** An agent that hands back an ambiguous case is working, not failing. If that number is zero, your agent is overconfident. **What does a failure cost?** A wrong draft costs a minute. A wrong payment costs a relationship. The same accuracy number means completely different things either side of that line. ## Who owns this The quiet organisational problem is that this work has no obvious home. It isn’t quite engineering, because judging whether an outcome is good needs domain knowledge. It isn’t quite the business, because building and running the evals is technical. In the teams where this works, someone who understands the actual work owns the eval set — they decide what “good” means and they review the failures — while engineering owns the harness that runs it. When nobody owns it, the eval set gets written once during the build, never updated, and the agent stops being measured on the day it starts mattering most. An agent isn’t finished when it works in a demo. It’s finished when you can say, with evidence, how often it works and how badly it fails when it doesn’t. If your team is putting agents into real workflows and working out how to tell whether they’re actually doing the job, that’s the kind of thing we help teams get right. [Talk to us](/enquire) if it’s useful, or [see how we run it in-house](/for-teams).