Self-Hosting an AI Agent: What to Get Right Before It Holds Any Credentials

To self host an AI agent safely, provision it in a fixed order: pin the image by digest, set the command the image actually expects, run it as a non-root user, bind every port to loopback, switch the safety rails on, and prove the container stays up for a full minute. Only then does it get a credential.

This is the order we settled on while building our own AI operating system on a single VPS in July 2026, using the official Hermes Agent image. It is not a troubleshooting post. The runtime failures and the post-incident debugging are in Hermes Agent in production. This one stops at the exact moment the first credential is issued, because everything before that moment decides how bad the first mistake can be.

What does self hosting an AI agent involve for a small business?

Self hosting an AI agent means running one long-lived container on a server you control. That container holds the model connection, the messaging gateway (Telegram, WhatsApp, a web dashboard), a cron scheduler, and eventually every API key the agent uses to do real work. The provisioning pass is the window when it holds none of them yet.

Our setup on 24 July 2026 was one Compose project, one container, one bind-mounted data directory, and a control-plane board the agent would later work from. Memory after boot was about 2.6 GB free on the box. Nothing about the hardware was interesting. What mattered was the sequence, because two of the steps only protect you if they happen before the container ever runs with a real brain.

If you have not yet picked an agent runtime, our Hermes Agent install guide covers the choice and the business use cases. The rest of this post assumes you have an image and a server and are about to write the Compose file.

What order should the provisioning steps run in?

Six steps, in this order: pin the image by digest, write the Compose file with the gateway command and a non-root UID, bind ports to loopback, set the safety rails and a placeholder model provider, boot and hold the container for sixty seconds with a restart count of zero, and only then hand it a credential.

• 1. Pin the image. Pull by tag once, record the digest, and reference the digest in Compose from that point on.
• 2. Write the Compose file. Explicit command, explicit user, restart policy, loopback-prefixed ports, one data mount, timezone.
• 3. Bind to loopback and verify it from the host with a socket listing, not by reading the Compose file back to yourself.
• 4. Set the safety rails in the agent's config before first boot: manual approvals, cron denied, secret and PII redaction on, and a model provider that physically cannot answer.
• 5. Boot, then watch the restart count for a full minute. Twelve seconds is not a stability check, for a reason covered below.
• 6. Arm it. Issue the first credential, which in our case was the model provider login. That step belongs to the parent post.

What should the Compose file look like before the first boot?

The Compose file needs five things set explicitly: an image reference by digest, a command that runs the gateway in the foreground, a non-root user, a restart policy of unless-stopped, and port mappings prefixed with the loopback address. Leave any of them to the image's defaults and you inherit a decision you did not make.

Ours is a single service. We describe the lines rather than paste the file, because the file also carries paths that map our host layout.

• image: the Hermes Agent image referenced as name@sha256 followed by the digest we recorded at pull time, not name:latest
• command: hermes gateway run, as a JSON array so no shell wraps it
• user: a high, unprivileged UID. The image defaults to 10000 through its HERMES_UID and HERMES_GID variables
• restart: unless-stopped, so the container comes back after a host reboot but stays down if we stop it on purpose
• ports: two mappings, the API port and the dashboard port, each prefixed with the loopback address so nothing binds on a public interface
• volumes: one bind mount for the agent's data directory, owned on the host by the same UID the container runs as
• environment: TZ set to the timezone the business runs on, so cron and logs agree with the humans reading them

Two things are deliberately absent. There is no Docker socket mounted, because a socket mount turns the container into root on the host and undoes the point of the non-root user. And there is no model API key in the environment block, because the first boot is meant to happen with no brain attached.

The upstream project's own docker-compose.yml is worth reading before you write yours. As of September 2026 it sets the command to gateway run, defaults the UID to 10000, uses host networking, and binds the dashboard to localhost with a comment explaining why. We wrote our first Compose file from the image alone rather than from that file, which is how we walked into the next section.

Why does the container exit immediately when started headless?

On the official Hermes Agent image, the container's CMD is the real main program, and the default CMD is the interactive chat interface. Started without a terminal, that program exits at once, so the container exits at once. The s6 supervisor inside the image does not keep the agent alive. That is the image's deliberate design, not a bug.

We found this by reading the image's own s6 run-script comments. The main-hermes service under s6 is a no-op that sleeps forever, and the comment above it says the container exits when the program exits. Our Compose file had no command line, so the image fell back to the chat TUI, which has nothing to attach to on a server and quits. Docker then restarted it, it quit again, and the cycle continued at a pace we will get to.

The fix is the single line the upstream Compose file already carries: run the gateway in the foreground. The image documents this as the recommended Docker mode. It starts the messaging platforms, the cron scheduler, and the board dispatcher, and it does not return, so the container stays up.

Before we understood any of that, we had spent an evening diagnosing why our OAuth login poller kept dying, and had concluded it needed a TTY. It did not. That misdiagnosis is the subject of the parent post's opening section, and we will not retell it here beyond the one number that matters: RestartCount reached 342 before anyone checked it.

Why run the agent as a non-root user?

A self hosted agent executes commands on your behalf, so the container is the isolation boundary. Running the process as an unprivileged UID means a prompt injection, a bad skill, or a plain mistake by the model runs with that UID's permissions inside the container, not root's. Combined with no socket mount, escaping to the host requires a kernel bug rather than a config oversight.

Two practical consequences follow. First, the bind-mounted data directory must be owned by that UID on the host before the first boot, because Docker creates a missing mount point as root and the agent then cannot write its own state. Second, the agent's terminal backend should stay in-container. Hermes supports several execution backends, and the moment you point one at the host, the UID protection stops meaning anything.

How do you bind to loopback only, and how do you prove it?

Prefix each port mapping in Compose with the loopback address, then confirm from the host with a listening-socket listing that the bound address is loopback and not the wildcard. The Compose file tells you what you asked for. The socket listing tells you what you got.

The verification we ran, and now run on every agent container, is two commands and one external check.

• On the host, list listening TCP sockets with the process name shown, and filter for the API and dashboard ports. The local address column must show the loopback address for both. A wildcard address there means the port is public.
• Inspect the container's port bindings with docker inspect and read the HostIp field for each mapping. It should match the loopback address, not be empty.
• From a machine outside the network, attempt a connection to each port on the server's public address with a short timeout. Both attempts must fail. If one succeeds, stop and fix it before anything else.

The dashboard is the port people forget. On our first boot it had no authentication provider configured yet. A dashboard with no auth on a public interface is a login-free admin panel for whatever the agent can do, which at that stage was nothing, and a week later would have been everything. Bind it to loopback and reach it over an SSH tunnel until auth is wired.

The upstream Compose file uses host networking, so there are no port mappings to prefix and the application itself decides what to bind to. The socket listing is then the only check that counts.

Which safety rails must be on before the first real boot?

Four config values and one placeholder, all set and read back before the container ever runs with a model attached: approvals set to manual, cron mode set to deny, secret redaction on, PII redaction on, and a model provider that points at a port where nothing listens. That last one is what lets the container boot at all without being able to think.

The placeholder provider deserves explanation. Current builds of the image refuse to start when no model provider is configured. Older builds idled inert. The new first-run check exits the main program, which under the CMD design means the container exits, which means a restart loop. The image's own documented non-interactive path is to set a custom provider with a base URL on the loopback interface at a port nothing is listening on. That satisfies the check and physically cannot produce an answer. We wrote the rule down as: deploy inert means placeholder provider now, real brain at arming.

The four rails are set with the agent's config command and verified with the matching get. Verifying is not optional. A rail that exists only as a line you believe you added is a declaration, and the theme of our entire build log is that declarations are not enforcement. Read each value back and keep the output.

• approvals.mode = manual. Every consequential action waits for a human until we later narrowed the boundary deliberately.
• cron_mode = deny. No scheduled job can exist until someone creates one on purpose.
• redact_secrets on. Tokens do not appear in logs or transcripts.
• redact_pii on. Same treatment for personal data, which matters once the agent reads mail and CRM records.

Why before first boot rather than after? Because the first thing an agent does when it has a brain and a channel is act. If a real provider is attached and approvals are still on the default, the first tool call runs with nobody watching.

This is the provisioning discipline we bring to every agent we build for clients through our AI agent development service. The container shape is the same whether the agent is answering a clinic's WhatsApp or reconciling a founder's inbox.

Why pin the image by digest instead of a tag?

A tag is a pointer that the publisher can move. A digest is the content hash of one specific image and cannot change. Docker's own documentation puts it plainly: pulling by digest lets you pin an image to that version and guarantee that the image you are using is always the same. For an agent that will hold credentials, that guarantee is the point.

We learned the cost of the alternative inside the same week. The image's behaviour changed between builds: older builds idled quietly with no provider configured, and the newer build we pulled exits on the same condition. Had we been on a tag, a routine restart could have pulled a different image and changed the failure mode under us mid-diagnosis. On a digest, the thing we were debugging held still.

The trade-off is real and Docker states it in the pull reference: pinning means Docker will not pull updated versions, which may include security updates. That is acceptable for an agent runtime as long as re-pinning is a deliberate, logged act. Our rule is that a digest change is a change request, reviewed like a code change, never a side effect of a restart.

Why does a 12-second stability check pass a crash-looping container?

Because Docker does not consider a container successfully started until it has stayed up for ten seconds. A container that lives for twelve seconds between crashes crosses that line on every cycle, so the restart back-off keeps resetting to its minimum, the loop never slows down, and any check that runs inside one of those windows sees a healthy container.

The mechanic is documented in two places. Docker's page on starting containers automatically says a restart policy only takes effect after a container starts successfully, and that starting successfully means the container is up for at least ten seconds and Docker has started monitoring it. The docker run reference adds the back-off: an increasing delay, double the previous and starting at 100 milliseconds, is added before each restart, and if a container is started and runs for at least ten seconds the delay is reset to 100 milliseconds.

Put those two sentences next to our incident. Our container was living between eight and twenty seconds per cycle. Every cycle that passed ten seconds counted as a successful start and reset the back-off, so instead of the loop slowing to minutes between attempts, it stayed tight all evening. RestartCount reached 342. Our stability check had been to wait twelve seconds and run a couple of board commands. It passed because it landed inside an up-window, and it would have passed most of the time.

The consequence was every OAuth poller we started inside the container dying seconds later, because each restart kills every process exec'd into the container. We burned about eight device codes and blamed the terminal before anyone read the restart count. The part the parent post does not make explicit: a short check is not merely weak, it is structurally blind to this failure, because Docker's definition of success is shorter than the cycle.

A stability check that means something looks like this.

• Read RestartCount with docker inspect. It must be zero, or whatever it was before your last deliberate restart.
• Read it again sixty seconds later. It must be the same number. We hold for sixty seconds minimum, which is six times Docker's threshold and three times the longest up-window we saw.
• Read StartedAt from the same inspect output and confirm it has not moved between the two reads.
• Watch docker events filtered to the container during the hold. Any die or start event during the minute is a fail, whatever the status column says.

Our exit criterion, written into the log at the time, was restarts equal to zero held for sixty seconds or more. It has not needed changing.

When is the container ready to hold its first credential?

When every one of the following is true at the same time and has been read back rather than assumed: the container has held a zero restart count for a minute, no port answers on a public interface, the four rails read back as set, the placeholder provider is in place, and the container has survived a host reboot. Then, and not before, issue the first credential.

Those were our Wave 0 exit criteria, and the reboot test is the one people skip. Unless-stopped only brings the container back if Docker itself starts on boot, so the test is a real reboot of the host, not a docker restart.

For us the first credential was the model provider login, done through a device-flow OAuth on a container that had been put into a maintenance mode so it could not crash mid-flow. The details of that arming pattern, and the model-id hunt that followed, are in the parent post. What happens after arming, meaning how you know the agent is actually working once it is, is in our post on AI agent observability. The full story of what the agent mesh became is in the Neogen AI OS case study.

Frequently asked questions

Should I use restart: always instead of unless-stopped?

Use unless-stopped for an agent. Both policies bring the container back after a host reboot. The difference is that always will also restart a container you stopped on purpose the next time the daemon starts, which is exactly what you do not want during an incident or a maintenance window. Unless-stopped respects a deliberate stop and still survives reboots.

Does a Docker healthcheck catch the crash loop described above?

Not on its own. A healthcheck changes the health field in the container's status, but a container that exits and restarts every twelve seconds never runs long enough for the check to matter, and the restart policy does not consult health at all. The restart count is the signal. A healthcheck is still worth adding for the slower failures that come later, where the process stays up but stops answering.

Do I need the dashboard port reachable from the internet?

No. The dashboard is an admin surface, and on a fresh install it may have no authentication provider configured. Bind it to loopback and reach it over an SSH tunnel. When you later want colleagues to use it from their desktops, put it behind a reverse proxy that handles auth and TLS, which is what we eventually did and is a separate project from provisioning.

My agent is already running on a tag. How do I move it to a digest?

Read the digest of the image you are running now with docker inspect on the container's image, write that digest into the Compose file in place of the tag, and recreate the container. You end up on exactly the image you were already running, so nothing changes except that it can no longer change without you. Do it before the next restart, not after.

How long does this provisioning pass take?

With the order above and the image already pulled, about an hour, most of it the sixty-second holds and the host reboot. Our first pass took an evening because we did the steps in the wrong order. This post exists so yours takes the hour.

The primitive

A container earns its first credential by staying up, and staying up means a restart count held at zero for a full minute, read twice, not a green status line that happened to land inside a twelve-second window.

If you are about to give an agent access to your inbox, your CRM, or your bank feed and want the provisioning pass done by people who have already made these mistakes on their own infrastructure, talk to us.