AI Agent Evaluation: When the Test Double Is All You Tested

AI agent evaluation that runs only against test doubles tells you the doubles work. On 25 August 2026 we shipped a WhatsApp reply-alert feature for JARVIS, our founder's agent, with 49 tests passing and a verified live deploy. The first real reply vanished. So did the second. The feature could not have fired a single alert, and every one of those tests stayed green.

The short version: our fixtures described the input we expected, not the input the system produced. The command-line tool framed its output differently, WhatsApp addressed the reply differently, and the quiet path logged nothing. One end-to-end call against the real subprocess caught the first defect before a phone was involved. What we changed is where fixtures come from.

What does AI agent evaluation need to test that unit tests cannot?

The boundaries where your code meets something you did not write: the model's CLI output, the messaging platform's address format, the log you will read on a live box. Unit tests with fakes cover your logic between those boundaries. They say nothing about whether the fake at each boundary resembles the real thing, and that is where agents break.

Martin Fowler's definition, borrowed from Gerard Meszaros's xUnit vocabulary, is useful because it is so plain: "Test Double is a generic term for any case where you replace a production object for testing purposes." A replacement is a claim that two things behave the same way. In our suite, nobody had tested the claim.

For a deterministic library you own, a hand-written double is usually faithful enough. For an agent, the things behind the doubles are an LLM subprocess, a third-party messaging protocol and a production log. All three change shape without telling you, and none of them read your spec.

What did we ship, and what did the 49 tests actually cover?

JARVIS could send WhatsApp messages to anyone our founder named, but inbound was locked to a single number. Every reply a recipient sent back went into a drop path. Outbound worked; the answer to it did not exist.

The design kept inbound closed and added a one-way alert. A successful send records the recipient in a 30-day ledger. When a ledgered recipient replies, the message goes to an isolated triage call with no tools, no history and no session, which returns ignore, attention or urgent plus one sentence. Only attention and urgent produce an alert to the founder. Nothing is ever sent back to the contact, and a reply can never become an instruction to the agent. The guardrails around it:

• Rate limits of 3 alerts per contact and 20 overall in any 60 seconds.
• Triage failures fail closed: no alert, no forward.
• Unknown senders, expired recipients, groups, status updates, the agent's own echoes and the founder's own number all keep the old drop path untouched.

The suite ran 49 tests against the real processReplyAlertUpsert boundary, the function the bridge calls on every inbound message. That phrase, real boundary, was why we trusted it. The function was real. Everything it called out to was not. The triage subprocess was a fake execFile returning bare JSON, and the inbound messages were built by hand in the address format we expected.

The deploy was asserted against the live health endpoint, and the rollback was rehearsed on the live box and passed. The log entry we wrote that morning listed exactly one open item: no live smoke test yet.

What happened when the first real reply arrived?

Nothing, which is the worst available result because it looks like success. Our founder supplied a secondary number the same day, JARVIS messaged it, and the number replied twice. No alert arrived and the box showed no errors. Both replies had simply gone.

Two defects were responsible, and a third problem made the first smoke test impossible to read. None of the three was reachable from any test in the suite.

Why did the triage parser reject every real reply?

Because it parsed the output we imagined. The parser ran JSON.parse over the whole of the subprocess's stdout. The real CLI, hermes chat -Q, prints the JSON object, then a blank line, then a session_id banner. Real output never parsed, not once.

The parse threw, triage returned INVALID_OUTPUT, and the fail-closed path did exactly what we built it to do: it swallowed the failure and sent nothing. The safety property and the defect produced the same observable behaviour, which was silence. No alert could ever have fired.

Every test fed the parser a fake execFile that returned bare JSON. The fake reproduced the interface of the subprocess, its call signature and callback, but not its output. So the 49 tests established that our code handled the JSON we had typed into a fixture. They could not have established anything else, however many of them we wrote.

The fix extracts the first balanced JSON object from stdout. The contract check after it is unchanged, so anything that is not exactly {priority, summary} still fails closed. We did not loosen the gate to let real output through. We fixed the step that read it.

Why did replies not match the number we had messaged?

Because WhatsApp no longer reliably addresses inbound messages by phone number. The reply arrived from a linked identity, a fifteen-digit identifier ending in @lid. Our alert branch demanded <digits>@s.whatsapp.net and matched the ledger on that. Every real reply fell through to the ordinary path and was dropped as an unauthorised user.

The spec had assumed the inbound remoteJid would be the same phone JID we sent to. That assumption lived in the design document, then in the fixtures, then in 49 green tests. It never lived in WhatsApp.

The answer had been on disk the whole time. Baileys, the open-source WhatsApp Web library the bridge runs on, had already stored the mapping from that linked identity to the phone number when the conversation paired. The bridge now resolves the sender through key.remoteJidAlt and falls back to that mapping store.

Resolution is the part we were most careful with, because an address resolver is an easy place to hide a permission by accident. It maps an address to a number and grants nothing. The resolved number is still tested against the ledger, so each of these stays ineligible, and each now has its own test:

• an unknown linked identity with no stored mapping
• a linked identity belonging to someone JARVIS never messaged
• a linked identity that resolves to the founder's own number
• a resolver that throws

Why could the first smoke test not tell us anything?

Because the verdict we expected most often logged nothing. Attention and urgent verdicts logged. Suppression by the rate limit logged. Triage failures logged. A reply triaged as ignore fell through without a trace.

So an empty log after a test reply was equally consistent with the feature working perfectly and with the branch never executing at all. From the box, those two states were identical. Dismissals now emit reply_triaged_ignored, so every reply that reaches triage leaves a line whatever the verdict.

This is the one we would have been slowest to find by ourselves. It breaks nothing. It only removes the ability to tell a working system from a dead one, which is the same shape as the six days our agent mesh was down while every health check stayed green.

We build agents for other businesses on the same gateway and bridge stack that runs our own agency. If you want reply handling, triage and fail-closed gates built by a team that publishes where its own tests lied, our AI agent development services page covers how that engagement works.

What changed about how we source test fixtures?

Fixtures now come from the real system, not from our description of it. We treat a fixture as a recorded fact about a dependency, and a hand-typed fixture as an unverified claim that needs a date and a source before it can back a test.

• Capture before you mock. For any subprocess, we run the real command once inside the target container and save its raw stdout, banner and blank lines included, as the fixture. The parser test reads that file. With this rule in place on 25 August, the session_id banner would have been in the fixture from the first commit.
• Record inbound shapes from the live channel. Address formats, message envelopes and platform metadata come from a real message captured on the box, not from documentation and not from our spec. The spec is where the phone-number assumption lived.
• Make one end-to-end call per external boundary before any human test. The cheapest test we ran all day was a single call to the real triage subprocess in the container. It caught the parser defect before anyone picked up a phone.
• Evaluate the real model on hostile input, not only on the happy path. We ran triage in-container on three cases: an urgent message came back urgent, casual chatter came back ignore, and a prompt-injection attempt came back as attention, reported to the founder as data and not obeyed.
• Log every branch, including the boring one. If a verdict can happen, it leaves a line.

The suite went from 49 tests to 56 that day, all passing. The count matters less than where the new tests point: at recorded real output and real addresses, instead of at what we believed they would be.

Does this mean the testing pyramid is wrong for AI agents?

No. It means our pyramid had no top. Mike Wacker's post on the Google Testing Blog puts it this way: "As a good first guess, Google often suggests a 70/20/10 split: 70% unit tests, 20% integration tests, and 10% end-to-end tests." We shipped with 49 unit tests and zero tests that crossed a real boundary. That is not a pyramid. It is a foundation slab.

The same post argues that end-to-end tests are slow, flaky and should be few, and we agree. We still run very few. Our case for the ones we keep is narrower than coverage: for an AI agent, unit tests are only as honest as the fixtures under them, and the only way to keep fixtures honest is to regenerate them from the real system. A handful of real calls is what keeps the rest of the suite truthful.

What was still unproven when we closed the log entry?

The alert itself. By the end of 25 August we had fixed both defects, verified triage against the real subprocess and deployed the corrected bridge. We had not yet watched an actual alert arrive from an actual reply, and we were waiting on two more replies from the test number. We recorded that as open instead of calling the feature done, because a green suite had already said done once that day.

The same morning produced one more false pass, in the deploy script rather than the feature: a health gate that matched the word connected inside disconnected. That one has its own write-up on how a substring check verified a failed deploy. Across the whole day, every false result came from our verification. The system did exactly what it had been told each time.

The primitive: a test double proves the double works

All three failures sat exactly where a fake stood in for reality: the CLI's real stdout framing, the real inbound address form, and the real logging needed to read a live run. A suite built entirely on doubles can be complete, green and blind at the same time.

The working form of the rule: for every double in an agent's test suite, name the real thing it replaces and the date its fixture was last captured from that thing. If the answer is that someone typed it, the test is checking your assumptions about the input rather than the input. It is the testing version of the argument in stated controls are not controls: a check that has only met a description has not met the system. The agent mesh this feature runs inside is written up in the Neogen AI OS case study.

Frequently asked questions

Should you stop mocking the LLM in agent tests?

No. Mocked model calls keep unit tests fast, cheap and deterministic, and you need that for logic like rate limits and ledger expiry. Keep the mocks, but build them from recorded real responses, and add a small set of tests that call the real model on fixed inputs, so a change in output format fails a test instead of a production run.

How is evaluating an AI agent different from evaluating the model?

Model evaluation asks whether the output is good: accurate, correctly classified, safe. Agent evaluation also asks whether that output survives the plumbing around it. Our triage model was never the problem. The agent failed anyway, because the parser, the addressing and the logging around a correct answer were each built on an assumption.

How many end-to-end tests does a small agent need?

At minimum, one per external boundary it depends on: each subprocess or model call, each inbound channel, each place a human reads the result. For the reply-alert feature that meant three. Run them before any human test, since the cheapest one we had caught the first defect without anyone touching a phone.

Does fail-closed design make bugs harder to find?

It can, and here it did. A fail-closed path turns an unexpected error into a quiet non-event, which is right for safety and wrong for diagnosis. The answer is not to fail open. It is to make the closed path loud in the logs, so a swallowed failure is visible to whoever operates the system even though it reaches nobody else.

Is this problem specific to WhatsApp?

The address defect is. WhatsApp's use of linked identities will meet any bridge built on the older assumption that a reply comes from the phone number you messaged. The parser and logging defects are general: any agent that shells out to a CLI or model and parses what comes back has the same exposure, whichever channel it answers on.

Want your agent's tests checked against the real thing?

If you run agents or automations where a passing suite is the main evidence that they work, we will go through which of your tests meet a real boundary and which only meet a fake. Talk to us and bring the test suite.