AI Agent Monitoring: The Checks That Never Fired

AI agent monitoring is how you detect that an autonomous agent has stopped working, changed something it should not have, or drifted from the state you expect. Most of it fails for one reason: the check exists, it is documented, and nobody has ever watched it fire. Across three incidents in August 2026 we hit that same fault three separate times.

This is a build-log post from our own AI operating system, where a mesh of agents runs real jobs against live systems. The incidents below are ours, with dates. None of the three was caught by the monitoring we had already written. All three were caught because somebody compared a printed number against what it should have been.

What is AI agent monitoring?

AI agent monitoring is the set of checks that tell you an agent has stopped producing work, taken a destructive action, or diverged from its expected configuration. It differs from ordinary uptime monitoring because an agent can be running, healthy and responsive while doing nothing useful at all.

We wrote about that first failure shape separately, after an agent mesh failed silently for six days. The processes were alive, so every health check was green, and the answer there was to monitor for silence rather than for errors.

This post is about the other half of the problem. Not what to watch, but whether the thing doing the watching works at all. A check has to fire on the right trigger, survive being read by a human, and have been seen to fire at least once. All three of ours failed a different one of those.

Why is an alerting path that has never fired not a control?

Because an untested alert has two independent ways to fail silently: it can miss the event, or it can detect the event and fail to deliver the notice. Both leave exactly the same evidence as a quiet week. Until you have watched a notice arrive on a real device, you have a design, not a control.

The context was our COO agent getting full autonomy over operations on 9 August 2026, and the question of what should sit underneath that autonomy. Counting the actions settled the design: 60 of the 68 actions available on our work-management platform are non-destructive, and only three genuinely destroy anything. Gating all of them would have put a human in the loop for sixty reversible things in order to catch three irreversible ones.

So the control became after-the-fact detection instead of pre-approval. For a principal who has said plainly that he does not want to approve things, that is the correct shape: a notice costs the recipient nothing when the news is good and reaches them in minutes when it is not. An approval gate costs attention on every occurrence, most of which are fine, which is how approval-by-habit gets trained.

Then we proved the path end to end rather than declaring it done. A phantom service, a phantom template and a fabricated node loss were injected into the stored baseline, so nothing real was touched. The run produced three notices and delivered all three. The next run produced zero, and the phantoms were gone from the snapshot. Detection, delivery and re-sync, each confirmed by observation.

Why does a drift detector that is always red get muted?

Because the correct human response to a permanently red check is to stop reading it. A detector that reports a problem on every run carries no information, and the person who mutes it is behaving rationally. The check is then worse than nothing, because the dashboard still claims coverage.

Ours was a generator that renders an org chart from personnel records. Its first version returned STALE immediately after regenerating its own output, which should be impossible. The cause was a seam: the function that carries forward the hand-written half of the document returns text that already begins with a newline, and the renderer appended it to a join that ended with another. Every run added exactly one blank line.

Left alone, that check would have reported drift forever, from the day it shipped. It was fixed by normalising the seam, then verified twice over: stable across two consecutive runs, and still returning a failure when a role actually changed. The second half of that verification is the part teams skip, and it is the half that proves the check still has teeth.

If you are putting agents into live operations and want this designed in from the start rather than retrofitted after an incident, that is what our AI agent development work covers.

Why do we diff inventories instead of reading logs?

Because a log only records the actions that went through the tool that writes the log. Comparing what exists now against what existed fifteen minutes ago catches the change whoever made it and by whatever route, including the route that bypasses your instrumentation entirely.

The immediate reason was availability: the platform exposes no audit endpoint to our key. Six different spellings of the obvious paths all returned 404. But a log would have been the weaker design even if one had existed.

The refusal list inside our own wrapper is a convention, not a boundary, and script execution is self-service for the agents that have it. Any agent can call the platform API directly in about three lines and never touch the wrapper that does the logging. An audit trail that an agent can step around is a record of the well-behaved actions only, which is the opposite of what you need it for.

Two failure modes were designed out explicitly, both of them repeats of mistakes already in this build log:

• The inventory read returns null on any failure, never an empty set. Writing an empty baseline after a failed read would have silently disarmed the check permanently, and every subsequent run would have reported all clear.
• The baseline only advances once the notice has actually been delivered. Committing the snapshot before the send would mean a failed send buries the deletion for ever.

The second one was already documented in a comment one function away from where it would have been reintroduced. Writing the lesson down did not prevent the repeat. Encoding it in the control flow did.

What should trigger an AI agent check?

Match the trigger to the cadence the work actually has, not the cadence you intend it to have. A check wired to an event that happens rarely, or only when a person remembers, is dormant during exactly the periods when mistakes accumulate unreviewed.

We were about to build this as a CI workflow that runs on push. Checking the repository state first changed the design: it was 17 commits ahead of origin and had not been pushed in five days. CI on push would have been asleep across the precise window in which every one of that week's mistakes was made. It would have reported green the whole time, because it would not have run.

So it became a local pre-commit hook instead. It takes about 1.3 seconds, needs no network, asks nobody for anything, and fires on every commit regardless of when anyone gets round to pushing. It is a machine checking a machine, silent unless something is broken. That distinction matters in an agentic system, where the reflex after an incident is to add another place a human has to say yes. Most of what went wrong that week needed detection, not permission.

The six invariants, and the incident behind each

Only binary invariants that had actually been violated that week made the cut. Anything needing containers, the live data store or the databases stayed in the self-test that runs on the server.

• Every host script parses, because a script that does not compile fails closed on the box, silently.
• No duplicate of a host script under an agent deploy directory, after a stale dispatcher that omitted one agent entirely sat one redeploy away from undoing live fixes.
• The approval gate sets no parse mode, after Markdown parsing silently dropped approval requests with an HTTP 400.
• No minted secret is ever returned to an agent, after a token was persisted to disk and printed back to the caller.
• No root verb is self-service, because root on the production box must never execute unattended.
• Generated skills document the parameters their handlers actually take, after one said body for a handler that takes content.

How do you prove a check actually works?

Break it on purpose and watch it fail. Introduce the exact fault the check exists to catch, confirm it fires, then repair the fault and confirm it goes quiet. A check that has only ever been observed passing has not been tested, it has been assumed.

We did that to all six invariants above. Four fired as expected. Two did not. The syntax check detected Python by shebang line, so a deliberately broken file with no interpreter line was skipped in silence and reported clean. The failure message on the skills check printed the literal string skills instead of the offending agent's name, from a single missing path call. Both were fixed and re-proven. The value of the exercise was not the four that worked.

This is the practice the Principles of Chaos Engineering, written by the Netflix engineers behind Chaos Monkey, define as "the discipline of experimenting on a system in order to build confidence in the system's capability to withstand turbulent conditions in production." Injecting a phantom deletion into a baseline is a very small version of the same idea, and it is affordable on day one of a build.

The alert-fatigue half is just as well documented. Google's SRE guidance on monitoring distributed systems argues that alerts which are not urgent and actionable train the people receiving them to ignore the ones that are. Our always-red drift detector was that failure in miniature, and it had not yet cost us anything only because it was caught in the same session it shipped.

The primitive: a check that only runs when someone remembers is not a check

The three incidents look unrelated until you line them up. A notifier nobody had watched deliver. A detector whose only honest response was to be muted. A CI trigger tied to a habit that had lapsed five days earlier. In each case the check was written, correct in its logic, and covered nothing, because the conditions under which it would actually run and be believed had never been examined.

The companion rule is shorter and does more work day to day: a check you have never seen fail is decoration. Both are cheap to apply. Breaking six checks on purpose took less time than writing any one of them, and it found two bugs in the checks themselves.

The wider build these incidents came from is written up in our AI operating system case study, and the same testing discipline applied to restores rather than alerts is in our note on backup testing. If you are running agents against live systems and cannot name the last time one of your checks fired, talk to us.

Frequently asked questions

How often should AI agent monitoring run?

Often enough that the window between a change and its detection is smaller than the damage that change can do in that window. Our inventory diff runs on a fifteen-minute cycle because the destructive actions it watches cascade quickly. Configuration drift checks run nightly, because drift is slow. Pick the interval from the blast radius, not from a default.

Do you need an observability platform for this?

No. Every check described here is a short script, a stored baseline file and a message send. The expensive part of monitoring an agent mesh is deciding what a correct state looks like and proving your check can tell. A platform will store and chart the signal, but it cannot tell you the signal was never wired up, which was our actual failure three times running.

What is the difference between monitoring and an approval gate?

An approval gate asks a person for permission before an action happens and costs them attention every time. Monitoring asks nobody anything and speaks only when something is wrong. Gates belong on the small number of actions that genuinely destroy data or move money. For everything reversible, detection after the fact is the cheaper control and the one that survives contact with a busy founder.

What should we monitor first when starting from nothing?

Start with absence rather than errors. An agent that has produced no output since Tuesday is the failure most likely to go unnoticed for days, because nothing in the system is complaining. Then add a diff of whatever your agents can irreversibly delete. Those two cover the failures that are both common and expensive.

Does this apply to n8n workflows as well as agents?

Yes, and the always-red trap is especially common there. A workflow with an error branch that fires on every run for a benign reason gets ignored within a week, and the genuine failure then arrives through a channel people have learned to skim. The same test applies: trigger the error path deliberately and confirm somebody notices.