AI Agent Memory: How Context Compression Cost Us Five Minutes Per Reply

AI agent memory failed on us in the most expensive way available: silently, and on every single message. On 31 July 2026 our agents were taking five minutes to answer a voice note. The speech pipeline accounted for 1.8 seconds of that. The remaining 300 seconds were a context compression call timing out.

Two unrelated faults were behind the same complaint - "slow, and not good". This is the chronology of both: what we believed, what the logs actually said, and the one-character config field that separated a 300-second timeout from a 1.2-second success.

What is AI agent memory, and what breaks first?

AI agent memory is how a long-running agent keeps a growing conversation usable inside a fixed context window. The standard mechanism is compression: summarise the old turns, free the tokens, carry on. What breaks first is not the agent forgetting things. It is compression taking longer than the reply it exists to enable.

Our agents run on Hermes, the runtime behind the Neogen AI OS build. Hermes triggers compression at 231,200 tokens. The session in question had reached roughly 236,000 tokens across 466 messages, so every incoming message crossed the threshold and every incoming message tried to compress.

Why did a five-minute reply have nothing to do with the voice pipeline?

Because the voice stack was 2% of the wall clock. We timed one reply end to end, 15:45:10 to 15:50:20, and split it by stage:

• Speech-to-text: 1.8 seconds
• LLM response: 5.5 seconds
• Context compression: 300.0 seconds, timed out

Total: 309.9 seconds. The two components anyone would suspect in a slow voice agent finished in 7.3 seconds between them. We had spent the previous session tuning the speech-to-text model and the voice provider, which is the sort of work that feels productive and would have moved this number by about two percent.

The 300.0 is the tell. Not 287, not 314. Exactly 300.0, seven times out of seven attempts that day - a 100% failure rate where each failure cost the full configured timeout before the agent was allowed to answer at all.

What does a failing compression call look like in the logs?

It looks like success with no progress. The line Hermes emitted after each timeout was:

Pre-API compression made insufficient progress: ~235,117 -> ~235,117 request tokens; skipping

The same number on both sides of the arrow. Compression ran, burned its entire budget, reduced the context by zero tokens, and the request went out at full size anyway. The agent then answered from an un-summarised 235k-token history, which is also why the replies were judged poor and not merely slow. Latency and quality had one shared cause.

The fallback chain was dead too, which is why nothing caught it. OpenRouter returned a payment and credit error. Nous returned "no Nous authentication found". With no working fallback, every message burned the full timeout rather than degrading to a slower-but-working path. An earlier session had reached 307,837 tokens, past the 272,000-token window of the model itself.

This is the failure mode that running Hermes agents in production keeps producing: nothing errors, nothing pages, and the system is simply 40x slower than it should be until someone times a single request by hand.

Why was the cheapest job routed to the most expensive model?

Because the compression provider was set to auto, and auto resolves to the agent's main brain. The same frontier model doing the reasoning was also being asked to summarise a 236,000-token history, inside a 300-second stream cap it could not meet.

Summarisation is easy work. It does not need the model you picked for judgement, tool use and multi-step reasoning. Defaulting the auxiliary path to the primary provider is convenient at setup and quietly wrong at scale, because the auxiliary call grows with conversation length while the primary call does not. The bigger the session gets, the worse the mismatch.

If you are wiring auxiliary calls into an agent stack and want the routing decided deliberately rather than by default, that is the kind of thing our AI agent development team sets up before the first production session rather than after.

How do you fix agent memory compression without changing the brain?

Point compression at a fast, cheap model declared as an auxiliary-only provider, and give it a timeout short enough that failure is survivable. We used Groq, whose key was already on every agent, running llama-3.3-70b-versatile. The main brain was not touched.

providers.groq: base_url https://api.groq.com/openai/v1, key_env GROQ_API_KEY, models [llama-3.3-70b-versatile]

auxiliary.compression: provider groq, model llama-3.3-70b-versatile, timeout 60

The timeout matters as much as the provider. Sixty seconds caps any future compression failure at a fifth of what the old one cost. We had no way of knowing in advance that Groq would answer in about a second, and the point of the cap is that we did not need to.

The direction is not ours alone. Anthropic's own context-management work reports that combining a memory tool with context editing improved agent performance by 39% over baseline, with context editing alone accounting for 29%, and a 100-turn web search evaluation completing workflows that would otherwise fail on context exhaustion while cutting token consumption by 84%. Treating memory as a first-class, separately-engineered subsystem is measurably worth it. Leaving it on auto is not.

Why did the first attempt fail on a config field the code itself documents?

We wrote api_key_env. The normaliser in Hermes documents api_key_env as a snake_case alias for key_env, so it looked correct and the software accepted it. On the auxiliary path it does not resolve. The client is constructed with a placeholder and returns 401:

named custom provider 'groq' has no resolvable api_key - request will be sent with placeholder no-key-required

Changing the field to key_env worked immediately. One underscore-delimited prefix, and the difference between a working fix and a silent 401 that would have kept every agent on 300-second timeouts while we congratulated ourselves for shipping.

This is the second instance of the same class we hit that week. A speech-to-text model override was likewise documented and likewise ignored on its code path. The pattern is worth naming: a documented alias is not a supported one. The documentation describes intent; only the code path that reads the field decides what is true.

How do you test an agent config change before it reaches every agent?

Write it to one agent, exercise the real code path against it without restarting, and only roll out on a confirmed success. Hermes reads config fresh on each load, so an auxiliary provider can be exercised with a genuine compression call while the agent keeps running. We got AUX-OK back from a single agent before the config went anywhere near the other three.

Our build log recorded the reason bluntly on the day: "Having twice this session assumed a config was valid because the software accepted it", the provider was written to one agent and exercised first. The 401 would otherwise have shipped to all four and silently kept the timeouts in place - a fix that looked deployed, changed nothing, and would have cost days before anyone re-timed a request.

One gotcha if you copy this technique. Environment variables from a .env file only reach the process environment after the agent module is imported, and key_env is resolved with a plain getenv. An ad-hoc test that checks the key before that import produces a false negative and will send you chasing a credential problem you do not have.

What was the second fault behind "slow, and not good"?

The agents were replying in two scripts at once. Asked a question in Malayalam, JARVIS answered with Malayalam words followed by Latin-script English in the same sentence. The text was correct. The voice was unintelligible.

The router's question was "does this string contain any Malayalam character?" On the strength of the first three words it sent the entire string, English remainder included, to the Malayalam voice, which cannot read Latin script. The round trip came back as noise, and the Malayalam half was destroyed along with the English.

Both earlier designs were wrong in the same way: they routed on the presence of a script rather than on the text being entirely one script. Mixed output was not an edge case, it was the single most common real input, and the router had no branch for it.

The fix was four-way routing, with mixed text normalised into Malayalam script in full and English terms transliterated phonetically before the Malayalam voice ever sees it. A verified round trip on "ente peru JARVIS" came back intelligible where the previous build returned nonsense.

We tried a single unified prompt first and rejected it. Asked to handle English, Manglish and mixed input at once, it translated pure English answers into Malayalam. The ambiguous English-versus-Manglish decision now happens in code, as a boolean pair, and the model is only invoked once the answer is already known. Give a model an ambiguous decision and it will resolve it confidently in the wrong direction.

What changed, measured?

Compression completed in 1.22s, 1.11s, 1.26s and 1.20s across the four agents after a single batched restart. A call that had a 100% failure rate at 300 seconds now succeeds in roughly 1.2 seconds, and replies that took around 310 seconds once a session crossed the threshold return in 10 to 15.

The quality half improved for the same reason the latency half did. Compression now actually succeeds, so the model stops trying to reason across 235,000 un-summarised tokens. One config block fixed both complaints because both complaints were one fault.

The script-level voice fix deployed with zero restarts, because the command provider spawns a fresh subprocess per call and replacing the script file takes effect immediately. Config changes cost a user-visible restart. Script changes are free. Knowing which of the two you are making is worth more than it sounds when four agents are mid-conversation.

Every timing here came from a stage-split of a single real request, not from a dashboard. If you want the general version of that argument, we wrote it up in how to monitor and observe AI agents.

Frequently asked questions

Does a larger context window remove the need for compression?

No, it moves the wall. Our earlier session hit 307,837 tokens against a 272,000-token window, so the larger window was already exhausted. Compression is what lets a session outlive the window; a bigger window only changes when compression first has to run. Long-lived agents need both.

Should compression use the same model as the agent's reasoning?

Almost never. Summarisation is undemanding work that grows with conversation length, while reasoning quality does not depend on it. Routing both to one frontier provider means your cheapest call inherits your most expensive model's latency and rate limits, and it degrades precisely when the session is longest.

How would we have caught this without a user complaint?

A per-stage timing breakdown on a sampled request, and an alert on any auxiliary call that hits its timeout more than once. Both were missing. Aggregate response time would not have helped: it was slow, and everyone already knew it was slow. The breakdown is what identified which of the four stages owned the 300 seconds.

What timeout should an auxiliary call get?

Short enough that a total failure is cheaper than the work it was meant to save. Our compression timeout went from 300 seconds to 60. If the auxiliary call takes longer than the primary call it supports, the timeout is wrong regardless of whether the call usually succeeds.

Is starting a fresh session a valid workaround?

As immediate relief, yes - a new session starts below the compression threshold, needs no restart, and is available to a user the moment you tell them about it. As a fix, no. It transfers the cost to whoever is using the agent and destroys the conversation history that made the agent useful in the first place.

The primitive worth keeping

A documented alias is not a supported one. Test the config against the real code path, on one agent, before it reaches all of them.

Everything else here follows from that. The 300-second timeout was findable by timing one request by hand. The 401 that would have replaced it was findable by running one real compression call. Neither needed better tooling, only the refusal to treat "the software accepted it" as evidence that it worked.

If you are running agents in production and cannot currently say which stage owns your slowest request, talk to us - that breakdown is usually the first thing we build.