AI Agent Observability: Our Mesh Was Dead for Six Days and Every Check Was Green

AI agent observability is the practice of watching for the absence of expected work, not only for errors. On 8 August 2026 we found that our own agent mesh had been dead for six days. Roughly 2,975 consecutive scheduled runs had failed across three agents. Every health check in the stack was green, because nothing in the stack was counting.

The way we found out is worth its own paragraph. Our founder-facing agent had escalated a problem: it could not get root on one of our machines to install a skill for a peer agent. The report was accurate and thorough. It had checked every credential rather than assuming one was missing. What it never checked was the premise. The peer does not run on that machine, and there was already an approval-gated lane on the correct one that the same agent had used before. Verified thoroughly, in the wrong place.

Underneath that sat the outage. This is the post-mortem, with the numbers, the wrong turns, and the two signals we now watch. The wider system it describes is documented in our Neogen AI OS case study.

What is AI agent observability?

AI agent observability is the ability to tell from outside the system whether your agents are actually completing work. Conventional monitoring answers whether a service is responding. Agent observability has to answer whether anything finished. Those two questions come apart, and the gap between them is where a mesh can die quietly.

Rob Ewaschuk puts the goal plainly in Chapter 6 of Google's Site Reliability Engineering: "Your monitoring system should address two questions: what's broken, and why?" The same chapter gives the canonical set to measure: "The four golden signals of monitoring are latency, traffic, errors, and saturation."

All four of those measure something that happened. Latency needs a request. Errors need a failure someone reads. Saturation needs load. The most expensive failure in an agent mesh produces none of them, because it produces nothing at all. A mesh that has stopped working and a mesh with no work to do emit the identical signal: silence.

How can an AI agent mesh fail silently for six days?

Ours failed on a guard that was working correctly. On or before 2 August 2026 the global model behind our inference gateway moved a version, from gpt-5.5 to gpt-5.6-terra. Our scheduler refuses to run any cron job whose stored inference config differs from the gateway's current one, and it refused every single time:

"Skipped to prevent unintended spend: global inference config drifted since this job was created (model 'gpt-5.5' -> 'gpt-5.6-terra'), and this job is unpinned."

That guard is right. We would rather a job not run at all than spend against a model it was never tested on. But it fails the job forever and it tells nobody. Roughly 2,975 refusals later, three agents were completely inert, including all three of the inbox pollers they use to talk to each other. One agent spent five days waiting for a reply from a peer that could not hear it.

A correct refusal that is silent is an outage generator. A job that fails a thousand times in a row is, from the outside, indistinguishable from a job that had no work to do.

Why was every health check green?

Because every check we had was a presence check. Presence checks answer whether a component is there. Nothing we owned answered whether anything had completed. Each layer passed its own test and then handed off to a layer that died on arrival.

• The message bus API served 200s throughout.
• The dispatcher correctly detected pending work and correctly woke the workers.
• Container isolation held for the entire six days.
• The workers then failed on start, 100% of the time, and the failure was recorded where nobody was looking.

This is the practical difference between monitoring a service and observing an agent. A service that returns 200 is usually doing its job. An agent mesh returns 200 at every hop while doing nothing, because the transport is healthy and the work is not. The number that mattered, consecutive failures per job, existed in our database and was read by no process.

How do you date an outage nobody noticed?

Cross-check the timeline against a store that does not share the first one's retention policy. Our execution log keeps only the last 1,000 rows. Its earliest visible failure was 3 August, which looked exactly like a start date and was actually a retention boundary. We were confidently wrong by a day and a half.

The message bus settled it. The bus has no row cap, and no message created on or after 2 August had ever been completed. Six days, not five. Never date an outage from the oldest surviving row of a rolling log, because a rolling log cannot show you anything older than its own window and will not tell you that it is hiding something.

Why did an earlier partial fix make the outage harder to see?

One job hit this exact error on 4 August, was pinned to the new model, and completed every day afterwards. The other seven jobs in the same class were never swept. Someone fixed the instance in front of them instead of the class, and the surviving evidence then argued that the problem was solved.

That is the part worth stealing. Partial remediation is worse than none, because the job you would check by hand is now the one that works. When you fix an instance of a class failure, the first question is how many siblings share the cause, and the second is what sweeps them all at once.

All 36 scheduled jobs across four agents are now pinned to an explicit model. The founder-facing agent's own 27 jobs had been unpinned too. Its escalation poller survived purely by the accident of having been created after the model changed. The next version bump would have taken it as well, and then nothing at all would have been left to raise a hand.

If you are running agents in production and cannot answer how you would learn that one had stopped, that is the gap to close first. It is the question we build around when we do AI agent development for clients, and it is cheaper to answer before an incident than during one.

What should an AI agent watchdog actually watch for?

Absence, on a clock, from outside the system. Ours runs every 15 minutes on a systemd timer and looks for two things. Either one alone would have caught this outage on day one, which is the point of having two: they fail independently.

• Three or more consecutive failures on any single scheduled job.
• Any bus message sitting in queued or delivered for more than three hours.

Neither threshold is clever. Both are chosen against how long we can tolerate silence rather than against any property of the jobs, which is the same logic Google's SRE Workbook applies to alerting on service level objectives: pick the window from the damage the outage does, not from what the system happens to emit.

An absence check needs an expectation, not a threshold

The reason a normal alerting rule cannot catch this is that there is no metric to threshold. Nothing goes up. To detect absence you have to write down, somewhere the running system cannot edit, what you expect to happen and how often. Our scheduler already held that: 36 jobs with intervals attached. The watchdog reads the schedule as a contract and asks whether reality matched it, which is a different question from asking whether anything went wrong. Every agent stack has this list somewhere, in a cron table, a queue definition, a workflow trigger. If it is only implicit, nothing can check it, and a job that silently stops being scheduled leaves behind no record that it was ever supposed to run.

The design rules the incident forced on us

Alerts go out as plain text on a root-only credential. That rule came from a neighbouring bug in the same session: the approval prompts were sent with Markdown parsing on, and underscores are italic markers, so a script filename with underscores in it reached our founder with the underscores silently stripped out. An approval prompt whose text differs from what will execute is authorising something nobody read. Never apply markup to text a human is being asked to approve.

The watchdog rate-limits to one alert per subject per six hours and sends recovery notices when a job comes back. It also skips any job whose configuration changed after its most recent failure. We learned that the hard way: repairing eight jobs at once produced two immediate alerts about jobs that were already correct and had simply not come round again on their schedule. Reporting the illness after the cure is how people learn to ignore a watchdog.

There is a related trap in checks that only exist while someone remembers to run them, which we wrote about separately in a check that only runs when someone remembers.

Why should you never auto-retry a failed agent message?

Because on an agent bus, failed usually means refused, not broken. Three messages from one of our agents were sitting in the failed state, and the payloads were not faults. They were decisions, written out in full, and one of them read:

"Cannot process: this is founder-only HR progression and includes employee compensation information outside this agent's permitted access."

A second message recorded that the agent had searched shared storage, found HR documents it could technically open, and declined to open them because policy said not to. That is a boundary holding under load, which is the only evidence that a boundary is real. Retrying a refusal badgers an agent into re-refusing, and the day it stops refusing is the day the control quietly fails. We took the automatic retry we had just built back out the same day, for exactly this reason.

Three states look identical on a dashboard and mean completely different things:

• A failed job is infrastructure. Fix it.
• A failed message is usually a decision. Read it before touching it.
• A message stuck in delivered means genuinely nobody is home. That is your outage signal.

Our watchdog surfaces failed messages with the reason the agent gave, and never retries them. The distinction between a control that exists and a control that is doing something is the subject of stated controls are not controls.

What did we get wrong while diagnosing this?

Five things, in the order we got them wrong. Writing them down is not humility, it is the cheapest way to stop repeating them.

• We diagnosed three failed messages as a transient provider fault and built automatic retry on that theory. The evidence killed it within the hour: a fresh unparented resend failed identically while the polling job itself completed cleanly. They were refusals. The retry was removed the same day.
• We dated the outage from the capped execution log and were off by a day and a half until the bus was checked.
• We wrote a new rule into the founder agent's policy file that contradicted the rule sitting directly above it, telling the agent to interrupt immediately under a heading that restricted interrupts to a preset list. A new rule that fights an existing rule is worse than the gap it was written to close. It was rewritten to route through the next scheduled digest, with an explicit line saying it does not widen the interrupt list.
• We conflated two unrelated error classes. One of them only ever affected a single watcher job and never touched the bus messages it was blamed for.
• Two test submissions sent live approval prompts to our founder before being rejected. Testing a notification path notifies someone.

The primitive: monitor for silence, not just for errors

Almost every monitoring tool you will inherit is built around events. An error arrives, a threshold breaks, a probe fails, a log line matches. All of them are triggered by something happening. The failure that costs the most in an agent mesh is nothing happening, and absence has no event to hang a trigger on. You have to schedule the question yourself, on a clock that lives outside the system it is watching.

Four questions worth asking of your own stack this week:

• What is the longest a queue item can sit before something says so out loud?
• If a scheduled job stopped being run entirely, which alert fires, and how many hours does it take to fire?
• Does anything you own count consecutive failures, or does everything only look at the most recent run?
• For any component you migrated everything else away from, what would tell you it is still doing work? Ours was left out of an event-driven migration on 31 July and was the only agent still functioning when the others died. A migration by omission leaves an undocumented survivor, and its survival gets mistaken for design.

We run our agency on this mesh, so every one of these lessons cost us something before it was written down. If you are building agents into your own operations and want the observability layer designed in rather than retrofitted after a six-day outage, talk to us.

Frequently asked questions

How often should an agent watchdog run?

Ours runs every 15 minutes. The interval that matters is not the one that feels responsive, it is the ratio between your check interval and how long a silence can last before it costs you something. We can tolerate a few hours of a dead queue, so a 15 minute check with a three hour queue threshold is comfortably inside that. A mesh handling live customer conversations would need both numbers much lower.

Does any of this apply to n8n workflows rather than agents?

Directly. n8n's execution list shows you workflows that ran and failed, which is the easy case. A workflow whose trigger stopped firing shows you nothing at all, because there is no execution to display. Pair the built-in view with an external heartbeat that expects a ping on a schedule and complains when it does not arrive.

We use a hosted agent platform. Is this handled for us?

Partly, and only on the vendor's side of the boundary. A hosted platform will tell you its API is up and often that your run errored. It cannot tell you that a run which should have happened never got scheduled, or that a message your agent sent is still waiting for a peer that stopped listening. Absence detection sits in your half of the split responsibility.

How do you stop watchdog alerts from being ignored?

Three mechanics did the work for us: one alert per subject per six hours rather than one per failed run, recovery notices so a closed incident visibly closes, and suppression of any job whose configuration changed after its last failure. That last one is the least obvious and mattered most. Alerting about something you have already fixed is what teaches people to swipe the notification away.

Should the watchdog run outside the system it monitors?

Ideally, yes. A watchdog that shares a fate with the thing it watches goes quiet at exactly the moment it is needed. Ours runs on a host timer rather than inside any agent container, which is enough separation for this class of failure but not for a host-level one. We know that is the weaker position, and an off-host checker is on the list.