Hermes Agent in Production: What a Month of Running an Agency on It Taught Us

A Hermes agent on your laptop and a Hermes agent running your company are not the same object. We have run one continuously since 24 July 2026. It started as a single profile with no credentials and is now four agents on Telegram, seven profiles, 119 skills, 36 scheduled jobs and a governed path to money and infrastructure. This is what the first month broke, in the order it broke.

Running a Hermes agent in production means solving four things the quickstart does not cover: the container must run in gateway mode or it exits on boot, each agent needs its own container and its own UID or credential isolation is fiction, every cron job must be model-pinned or a provider change kills it silently, and every silent failure needs something that counts it.

Everything below comes from our own build log, not from documentation. Dates, error strings and counts are as recorded. The system it describes is the one we wrote up in the Neogen AI OS case study, and it is the same architecture we now deploy for clients as AI automation services.

What does running a Hermes agent in production actually involve?

Four layers, roughly in this order: a container that stays up, a brain that is authenticated, credentials scoped per profile, and a gate between the agent and anything consequential. Everything after that is content. Everything before that is a demo.

A month in, the shape of the deployment is:

• Four Telegram-facing agents across seven Hermes profiles, one container each for the ones that hold credentials
• 119 skills distributed by role, with the apex profile at 36 and the profile that executes real-world changes at 19
• 36 cron jobs, from twice-daily digests down to per-client invoice reminders
• 17 registered action types behind a one-time-code approval gate
• 1.4 GB of RAM for three agent containers on an 8 GB box, which is the number people ask about first and the least interesting one here

Why does the Hermes container exit as soon as you start it?

Because the image's default CMD is the interactive chat TUI, which exits immediately with no terminal attached, and the container is designed to exit when its program exits. The s6 service named main-hermes is a no-op that sleeps forever. The real main program is the CMD. Override it with gateway mode and the container stays up.

In compose terms, the fix is one line:

• command: ["hermes", "gateway", "run"]

This cost us most of a day, and the way it cost us the day is the more useful lesson. Every OAuth poller we started died within seconds. We tried nohup, tmux, setsid and a nested SSH session, and burned eight device codes. We diagnosed it as a TTY problem, because that is what "the process dies when I detach" usually means.

It was not a TTY problem. The container was restart-looping every eight to twenty seconds and each restart killed every process we had exec'd into it. RestartCount was at 342 by the time we looked. Our own stability check had been twelve seconds and a couple of board commands, and it passed only because it landed inside an up-window. Verify container stability before you debug process behaviour, over a window longer than the restart interval.

Deploy inert, arm later

Current builds of the image crash-loop when no model provider is configured, where older builds idled harmlessly. That matters if you want to provision a container before you hold credentials for it, which you usually do. The image's own documented non-interactive path solves it: configure a placeholder provider pointing at a port nothing listens on. The first-run check is satisfied and the agent physically cannot think. "Deploy inert" now means "placeholder provider, real brain at arming". We covered the wider provisioning path in our write-up on self-hosting an open-source AI agent.

Pin the image by digest, not by tag

We pinned the image by digest at pull time. An agent deployment is a set of behaviours you have tested, and a floating tag replaces those behaviours without telling you. This is also an s6-overlay image, which constrains what you can safely change about how it runs - see the s6-overlay documentation on user permissions before you touch it.

How do you complete an interactive OAuth login on a headless container?

Put the container into maintenance mode first. Pin the entrypoint to sleep infinity so nothing can crash, run the device flow with docker exec in detached mode writing to a log file, confirm the poller is holding that log's file descriptor open, hand the code to the human, then restore the real CMD. Any instability in the normal command otherwise takes the poller with it, and device codes expire faster than your patience.

The second trap on arming is the model identifier. A ChatGPT-account Codex OAuth credential rejects every model slug ending in -codex, with an error that reads like a plan restriction rather than a naming problem. Seven candidates from the public model caches all failed; the working slug was named in a comment in the provider source. What settled it was a probe loop rather than research: set the model, ask for an exact sentinel string, grep for it, break on match.

How should credentials be scoped across Hermes profiles?

One .env per profile, containing only the keys that role needs, and the restriction enforced inside the tool rather than in the prompt. Our SEO profile cannot see finance, ads, CRM, GitHub or Telegram credentials. The profile that executes changes to the outside world holds no write credentials at all.

The principle we ended up phrasing as "the profile is the permission" only holds if the enforcement is mechanical. A prompt instructing an agent not to write is not a control. It cannot be, because a prompt-injected agent has no way to distinguish an operator's instruction from injected text - which is why prompt injection sits at the top of the OWASP Top 10 for LLM Applications.

So the reductions are in the code:

• The ads reader issues GET only. It cannot POST even though the token carries management scopes.
• The Google Ads tool accepts SELECT-only GAQL against an allowlisted set of resources.
• The SEO data tool has an endpoint allowlist, a 25-task cap per run and a daily spend cap with real spend tracking, because that API bills per request.
• The finance tool holds a full read-write grant but refuses any write without an explicit confirmation flag, verified by trying it.

There is a Hermes-specific gotcha buried in this that cost us a real bug. Hermes loads a profile's .env for its own use, but it does not export those variables into tool subprocesses. Our data tools passed every test we ran, because we were passing the environment explicitly on the command line. When we finally checked the running gateway's actual environment, it had zero of the four variables we had wired.

Every tool the agent runs would have thrown a KeyError the first time the agent, rather than us, ran it. The fix is that each tool self-loads the profile .env through a shared loader, and we re-tested all of them with no environment passed at all. That loader later became the right place for a global socket timeout and a retry helper, so one change covered every tool - the earlier version had no timeouts anywhere, and one stalled provider was enough to hang a whole digest.

The operator inversion

The profile that changes the world is the one with the fewest credentials. Our ops profile has 19 skills and no write keys. It submits proposals to the approval gate and a human releases them with a one-time code. Capability without custody.

Does profile isolation stop one agent from reading another agent's secrets?

Not inside a single container. All our profiles ran as the same Unix UID with terminal, file and code-execution tools enabled. Directory permissions of 0700 isolated nothing, because one user owned all of them. Any agent could read any other agent's .env and secrets directory.

An external review raised it and we verified it in about a minute. The framing that made it urgent was not "one agent can read files" - it was identity theft. Running inside the HR agent's session using the apex agent's data key, a whoami against our access-controlled data store returned the apex profile with read permissions on everything. The host-side access control list did its job faithfully. It simply was not us asking.

This also forced a correction to something we had written down two weeks earlier. We had concluded that team members should get messaging channels only, on the grounds that per-profile credential isolation was fail-closed. That clause was wrong, and being wrong in writing is why nobody re-examined it. The agent holds a terminal regardless of which interface you talk to it through. Restricting the interface cannot contain a shared trust domain.

One container per agent, and the four things that break when you split

The fix is one container per agent, each bind-mounting only its own home directory, each on its own UID. That gives two independent layers: the sibling's path does not exist inside the mount namespace, and the host UID could not read it even if it did. We verified the second layer directly by attempting a read across containers and getting Permission denied.

Four things bit us during the split, all of them the same shape - a thing that was shared, quietly, that we did not notice we were copying:

• The image refuses a compose-level user: directive outright, because an arbitrary UID breaks its s6 supervision tree. Set HERMES_UID and HERMES_GID instead and let the image remap its own user at boot. An earlier test that appeared to work had bypassed s6 entirely by overriding the entrypoint, which is worth knowing before you trust a result like that.
• A profile directory is not a complete Hermes home. Each split home needs the shared auth credential and the relevant root config keys merged under it, or the gateway will not start.
• That merge copied the dashboard's session-signing secret into all three containers. A shared signing secret means a cookie minted by one dashboard would likely validate against another. Harmless while two of the three published no ports, fatal the moment they did. This was a bug we introduced while fixing a security problem, which is the normal way security bugs get introduced.
• The same merge enabled the API server in every home while its key existed only in the old root .env, which was not copied. All three gateways logged "Refusing to start" until each got its own unique key. The root .env and the root config file were a matched pair, and splitting one without the other breaks the gateway.

The rule that came out of it is short enough to enforce in review: never mount one agent's directory into another agent's container. That is the entire control. We go through the wider topology in our breakdown of agent mesh architecture.

Why did three separate Telegram bots all answer as the same agent?

A session-lookup bug in Hermes v0.19.0. The gateway namespaces session keys by profile, but the fallback branch of the recoverable-session lookup drops the profile name and matches on source, user ID, chat ID, chat type and thread ID only. For a Telegram direct message the chat ID equals the user ID, so that tuple is identical across every bot.

The symptom was that all three of our agents introduced themselves as the HR agent, and asking the founder agent about the CRM produced "that belongs to another agent". One account allowlisted on three bots had collapsed onto a single session, and that session belonged to whichever profile had been used most recently.

The blast radius was nil, because only one person was in more than one allowlist and that person is authorised for all of it. The latent risk was the actual problem: any subordinate added to a second bot's allowlist for convenience would have inherited that profile's credentials and access-control identity.

There is no configuration option for this. We checked the config schema and the CLI. The fix is upstream - add the profile name to the session lookup predicate and its index - and we filed it (#74285). Until then the operating rule is that a Telegram account appears in exactly one profile's allowlist, ever. If one person genuinely needs two bots, use a group chat, which has a distinct chat ID and does not collide.

The dashboard has no multi-user model

Related, and worth knowing before you promise a colleague a login: the Hermes dashboard's basic auth is global. There is no concept of allowed profiles, and the console and PTY endpoints expose a terminal inside the container, so any dashboard login can reach every profile's credentials. A reverse proxy pinning each subdomain to one profile does not fix it - roughly 140 endpoints accept a profile parameter, plus six websockets, and missing one defeats the scheme.

The supported answer is a separate Hermes instance per person. That can be another container on the same box behind one TLS proxy with per-agent subdomains, at roughly 1.2 GB each. You do not need another VPS.

How do you let an agent take real actions without giving it root?

Move the credentials and the approval mechanism to somewhere the agent cannot reach, and let the agent submit proposals rather than execute commands. Ours runs as a root-owned executor on the host behind an SSH forced command, with write credentials in a root-only file and a typed one-time code delivered to a human out of band.

Our founder set the policy in five words before any of it was built: "reads free, writes gated." Everything watching, reporting, flagging and proposing is autonomous. Everything consequential needs a code. The agent is never capability-limited, only consequence-limited.

Two insights shaped the implementation, and both are about where things live rather than what they say:

• An approval code stored inside the container is theatre. The agent can read it, or brute-force a short hash. The code and the write credentials must live where the agent has no path to them.
• Execution is registry-only. Ten registered action types at launch, seventeen now. Anything unregistered is refused at the door rather than evaluated on its merits.

The codes are single-use, bound to one specific action, expire in 60 minutes, and auto-reject with an alert after three wrong attempts. We tested the gate adversarially before trusting it: the agent cannot read the code through the status command, wrong codes are rate-limited, the credentials are unreadable, the SSH key yields no arbitrary command, and unregistered action types are refused. Then we ran the full submit, code, approve, execute, audit loop live.

A forced command is doing the heavy lifting here, and it is easy to get half-right. Scoping sudo alone is not enough - without the command directive in authorized_keys the key still yields an arbitrary non-root shell. The OpenSSH authorized_keys options are worth reading properly rather than copying from a blog post. Ours was caught by an adversarial review pass, not by our own testing.

One implementation detail that saves a lot of architecture: you do not need a second bot for the approval channel. Only one process may receive a given bot's updates, so a second receiver just gets a 409. But sending is unlimited. The gate sends its approval requests into the existing chat and the human replies with the code. One bot, one chat, no second identity to secure.

Then we over-restricted, and had to undo it

Having built the gate, we made the agent's own tools and skills read-only. That was reverted within a day, at the founder's insistence, and that was the right call. With the gate enforced server-side, freezing the agent's skills bought almost no security - editing a wrapper gains nothing when the wrapper cannot execute anything consequential - while disabling one of the platform's headline capabilities.

The principle: restrict the consequences, not the capabilities. Version-control the drift instead of forbidding it. Agent-authored skills get captured into a git repository so self-modification becomes a reviewable diff rather than an invisible change. An over-restricted agent is a failed agent.

If you want this architecture without spending a month discovering its edges, this is exactly what we build as an AI operating system deployment.

What breaks in week two that did not break in week one?

Silent failures. Week one breaks loudly - containers exit, gateways refuse to start, credentials 401. Week two is when something correct and quiet stops the system while every dashboard stays green. Every incident below had a working control at its centre.

A model change killed every unpinned cron job, and nothing said so

On or before 2 August the global model moved from gpt-5.5 to gpt-5.6-terra. Hermes refuses to run any cron job whose stored inference configuration differs from the gateway's current one, with this error:

"Skipped to prevent unintended spend: global inference config drifted since this job was created (model 'gpt-5.5' -> 'gpt-5.6-terra'), and this job is unpinned."

That guard is correct. It will not spend money on a model the job was not created against. But it fails the job forever and it tells nobody. We accumulated roughly 2,975 consecutive failures across three subordinate agents, including all three inter-agent message pollers. The entire subordinate mesh was dead for six days and looked idle.

Every layer was healthy throughout. The message bus served 200s. The dispatcher correctly detected pending work and correctly woke the workers. Container isolation held. The workers then died on arrival, one hundred per cent of the time. Our apex agent spent five days waiting for a reply from an agent that could not hear it.

Three lessons came out of that one incident, and the first is the named primitive of this entire month:

• A correct refusal that is silent is an outage generator. A cron job that fails a thousand times in a row is, from the outside, indistinguishable from a cron job that had no work to do. Nothing in the stack was counting.
• Never date an outage from the oldest surviving row of a rolling log. The execution database keeps the last 1,000 rows, so its earliest visible failure was a retention artifact and understated the outage by a day. We fixed the true start from an independent store with a different retention policy.
• Partial remediation is worse than none. One job hit this exact error four days in, was pinned, and completed every day afterwards. The other seven were never swept. Someone fixed the instance they were shown instead of the class, and the surviving evidence then looked like a solved problem.

All 36 jobs across all four agents are now pinned to an explicit provider and model. The apex agent's 27 jobs had been unpinned too. Its escalation poller survived purely by the accident of having been created after the model changed, and the next model bump would have taken it as well.

The scheduler reported healthy while every tick failed

A separate incident, same shape. docker exec defaults to root. Running a cron edit that way rewrote the jobs file as root-owned with 0600 permissions. The gateway runs as a non-root UID, so every subsequent tick died with a PermissionError and not a single job ran.

The scheduler kept reporting "Gateway is running, 27 active jobs" while failing every tick. We caught it because an overdue poller never fired, not because anything reported it. The command had returned success 27 consecutive times. The write succeeded; the readers lost access.

Verifying the edit is not verifying the system. Any change to a file that a service owns needs an after-check on the service, not on the write. And run your exec commands as the service's UID, not as root, which is a one-flag habit that removes the whole class.

Every restart interrupted whoever was mid-conversation

Users kept seeing "Gateway shutting down, your current task will be interrupted" during chats. We assumed a crash loop. RestartCount was 0 on every container, so Docker was not restarting anything, and the log named the sender as s6 rather than a fault.

The gateway start log, converted to local time, mapped one-to-one onto our own change windows. Every event was a restart we had issued ourselves. The aggravating factor was a default: the agent restart drain timeout ships as 0 despite an in-product tip claiming 60 seconds, so a restart interrupts in-flight work immediately, logging "drain timed out after 0.0s with 1 active agent(s)".

Set it to 60 and batch your configuration changes into one restart. Every restart is visible to whoever is mid-conversation, and the gateway start log is the audit trail to check before blaming the platform.

The approval prompt was not showing what would actually run

Our approval gate sent its Telegram messages with Markdown parsing enabled and unescaped content. Underscores are italic markers, so a script named generate_client_mou.py rendered to the approving human as generateclientmou.py - and rendered differently in the approval than in the failure notice, depending on how many underscores happened to pair up across the message.

An approval prompt whose text differs from what executes authorises something nobody read. Never apply markup to text a human is being asked to approve. Plain text, always. We now also refuse two approve-by-reference shapes at submit time: long encoded blobs, because base64 is bytes a human cannot see, and fetch-then-execute chains, where what runs is whatever the URL serves after approval.

A container path was submitted to a host executor, and left debris

The host executor runs on the host. The agent's data directory is a path that exists only inside a container. An agent submitted a script path under that container path to a host-root executor twice for the same document, burning two approvals and producing nothing.

The first attempt included mkdir -p, which created a phantom directory tree on the host mirroring a container path. A client document was subsequently written into it: outside every container, outside backup scope, unreachable by any agent. We confirmed by checksum that the container copy was byte-identical before quarantining the host tree.

Preflight the namespace, not just the syntax. Submissions referencing a container path without a docker exec are now refused with an explanation, as are interpreter targets that do not exist on the host.

Why did 86% of our approval requests turn out to be noise?

Because there was no lower write lane. Of 138 gated approvals in the fortnight to 8 August, 119 were requests to run an arbitrary host command and only 19 were registered business actions. The human approved 106 and rejected 4. A 96% approval rate is not a filter, it is latency - and 42 of those 106 then failed anyway.

The cause was not permissiveness. An agent that wanted to run its own script inside its own container had no mechanism to do so, so it asked for root on the host. If the only write lane is root, every routine task becomes a root request. That is privilege escalation by absence of a lower lane, and the fix is not to tighten the gate. It is to build the lane that should have existed.

The new action type executes a script inside the submitting agent's own container as its own UID. That is strictly less privilege than the host command it replaces, because the agent already owns that filesystem. Critically, the target container is derived from the SSH key identity and can never be passed as a parameter, or the lane becomes a lateral-movement path.

The other half of the fix was documentation. The hand-written skill describing the gate said, in effect, "you can do anything on the host through this verb" and never mentioned that most of an agent's work was already self-service. The agents were behaving exactly as instructed. That skill is now generated from the gate's own registry tables, because a document derived from the code cannot drift from it.

How do you know a Hermes agent is actually working?

You do not, unless something counts. Every incident above had healthy components and a dead system. The missing layer in all of them was not permission or isolation - it was anything that noticed a streak of failures and said so. Detection, not another approval gate.

Match the trigger to the actual cadence

We were going to build this as a CI workflow on push. Before writing it we checked the repository and found it was 17 commits ahead of origin and had not been pushed in five days. CI on push would have been dormant across exactly the window in which every one of that week's mistakes was made.

So it became a local pre-commit hook instead. Around 1.3 seconds, no network, no approval, fires on every commit regardless of when anyone pushes. A check that depends on someone remembering to do something is the same shape as an access-review report that ran on a timer, went red, and sat unread for a week.

It encodes only invariants that are binary and were actually violated. Not best practices: incidents. Every host script parses. The gate never sets a message parse mode. No minted secret is ever returned to an agent. Skills document the parameters their handlers actually take.

Then we deliberately broke each check to confirm it fires. Four did. One did not, which is the entire argument for doing this step. A check you have never seen fail is a check you are assuming.

A manifest check is not a restore

An external audit scored our OS 5 out of 10 and flagged that the backup had never been restore-tested. Fixing that justified itself immediately. The expanded archive was built, verified against its own manifest, encrypted and uploaded. Then the restore drill decrypted it and could not extract it.

Two paths, one a directory and one a file, shared a basename. Passed to tar as separate entries they both landed at the archive root under the same name. tar wrote both without complaint and refused to unpack them. The preflight had listed every required path and every path was genuinely present. The archive was simply not extractable.

The second finding was scope. The backup carried the agents, the data store, the audit trail and a proper database dump. It did not carry the dispatcher, the watchdog, the systemd units, the directory holding every write credential, or the message history that lived in a Docker volume rather than under a backed-up path. A backup of the data but not the control plane restores to files nobody can operate.

What we would do on day one if we started again

In order, before any content or skills work:

• Pin the image by digest and set the command to gateway mode in the first compose file you write.
• One container per agent from the start, each with its own UID and its own home. Retrofitting this is a security-sensitive migration; doing it on day one is a compose file.
• One messaging account in exactly one profile's allowlist, written down as a rule before anyone asks for a second bot.
• Every cron job pinned to an explicit provider and model at creation, plus a check that counts consecutive failures per job, and a restart drain timeout set to something other than zero.
• Credentials scoped per profile, with each restriction enforced in the tool. Give the executing profile no write keys at all.
• A restore drill before the first real record exists, so the drill is cheap when it fails.

Frequently asked questions

Can developers with server access see the agent's credentials?

Yes. Root on a box reads every plaintext credential on it, and file permissions are meaningless against root. Profile isolation protects agents from each other and never protects anything from a human with root. If your agents hold finance or HR credentials, that box must not be the box your engineers have root on. Those are two separate walls.

Does deleting credentials after an exposure fix it?

No. Only rotation does. We migrated an agent to a dedicated box and later deleted the old container and its data entirely, and none of that undid the exposure of the credentials that had sat on the original host. Deletion removes future access; it does nothing about anything already copied. Where rotation is declined, log it as an accepted risk with a name against it rather than treating the cleanup as a fix.

Can you give an agent skills written for a laptop?

Only with an environment note injected into each one. Skills written for a workstation assume a browser, a repository checkout and local command-line tools, none of which exist on a headless box. Ported without that note, they produce an agent that confidently claims to have rendered a video it never rendered. We add a block after each skill's frontmatter stating what that agent can and cannot do in that environment.

Why did testing say a skill was missing when it demonstrably worked?

Because the headless one-shot mode does not load the profile's skill index. Ask it whether it has a given skill and it answers no, including for skills that work perfectly through the chat gateway. We nearly "fixed" a deployment that was never broken. Verify skills through the gateway, not through one-shot invocations.

The primitive worth keeping

If one line survives from this month, it is the one our worst outage produced: a correct refusal that is silent is an outage generator. Every control in that stack worked exactly as designed. The system was dead for six days. Guards protect you from the failure they name and hide it from you at the same time, so anything that can refuse must also be counted.

The second is the one that keeps the system usable: restrict the consequences, not the capabilities. An agent you have crippled is an agent nobody uses, and an unused agent is the most expensive outcome available.

We are still running this, still logging it, and still finding things in week five we did not find in week one. If you are deploying agents against real operations and want the failure modes before you meet them yourself, talk to us about your build - or read how we approach AI agent development for clients.