AI Agent Architecture: The Agent Mesh That Runs a Real Company

AI agent architecture is the set of boundaries around an agent, not the instructions inside it. In our build, each of the four agents that run Neogen Media gets its own container, its own Unix UID and its own database role, and the database itself is not reachable from any agent container. That last part is the whole design.

The short version: one container and one UID per agent, a git-backed markdown store that lives on the host rather than inside any agent, a Postgres control plane where row-level security makes every filtering decision, and a write gate that holds the approval code somewhere the agent has no path to. The rule underneath all of it is that an agent must not be able to reach the thing that limits it.

Between late July and early August 2026 we went from one agent to four. Every structural change in that period came from a specific failure, including two occasions where we shipped a control that did nothing at all and only found out later. This is the chronology, with the wrong turns left in. The system it describes is documented in our Neogen AI OS case study.

What is AI agent architecture, and why does the prompt not decide it?

AI agent architecture is the containment design around an agent: which process it runs as, which filesystem it can see, which network it sits on, and which credentials exist within its reach. The prompt describes intended behaviour. The architecture decides actual capability. When the two disagree, the architecture wins every time.

We learned this from the write gate design before we had a mesh at all. Read-only by default, enforced by prompting, is convention rather than enforcement. A prompt-injected agent cannot tell its owner's instruction from injected text, so a rule that lives only in the system prompt is a rule the attacker also gets to edit. The OWASP Top 10 for LLM Applications ranks prompt injection first and excessive agency sixth, and those two entries describe the same failure from opposite ends: the model is convinced to want something, and the surrounding system was built to let it happen.

So the design question is never what the agent should be allowed to do. It is what remains true when the agent has been convinced to do the wrong thing.

Why did three agents in one container turn out to be one agent?

Because they ran as the same Unix user. Three profiles shared a container, all as UID 10000, each with terminal, file and code-execution tools enabled. The 0700 permissions on each profile directory isolated nothing, because a single user owned all three of them.

The practical consequence was that either of the two subordinate agents could read the founder profile's environment file, which held 47 secrets, plus every key in its secrets directory. That alone would be bad. What made it worse was the next step.

Our data store is served to agents through a reader that identifies the caller by key. Running inside that shared trust domain, holding the founder agent's data key, the reader reported the caller as the founder profile with read access to everything. The host-side access control list was correct and did its job. It simply was not us asking.

An agent that can read a sibling's credentials does not merely see the sibling's files. It becomes the sibling everywhere that credential is honoured, including on systems that were never part of the container at all.

The wrong sentence in our own build log

An earlier entry in our build log stated that team members would get messaging channels only, because per-profile credential isolation was fail-closed. That clause was wrong, and it is the reason the problem sat unnoticed.

The original finding had been scoped to the dashboard, which has no per-profile access control, and concluded that a chat-only surface was therefore safe. It is not. The agent holds a terminal regardless of which interface you talk to it through. Restricting the interface cannot contain a shared trust domain, and we had written the opposite down in a document we then trusted.

Correcting your own log matters more than it sounds. A wrong conclusion recorded confidently becomes the thing nobody re-examines.

How does one container per agent actually isolate them?

Through two independent layers. Each container bind-mounts only its own home, so a sibling's path does not exist in that container's mount namespace, and each runs on its own UID, so the host would refuse the read even if the path were there. We verified the second layer directly: one agent's UID gets permission denied against both siblings' environment files.

Either layer alone would be enough on a good day. The point of having both is that mount configuration is the kind of thing that gets edited by someone in a hurry, and UID ownership is not.

The gotchas we hit doing the split, in the order they cost us time:

• The runtime image refuses a plain user directive in compose, aborting on startup because an arbitrary non-standard UID breaks its s6 supervision tree. You remap through the image's own UID and GID environment variables instead, and it chowns the volume at boot.
• An earlier test that passed a UID flag directly appeared to work, but only because the same test overrode the entrypoint and bypassed s6 init entirely. A test that skips the init system is not testing the container you will run.
• A profile directory alone is not a complete agent home. Each home needs the shared auth file plus the root config keys merged under the profile's own, or you get a container that boots cleanly and knows nothing about itself.
• The no-new-privileges security option crash-loops s6-overlay images, because its init needs suexec to fix runtime directory permissions. The container healthcheck caught it. We documented it in the compose file so nobody re-adds it on a future hardening pass.

After the split, only the founder agent publishes ports, and only on loopback for the desktop app. The other agents publish nothing: their dashboards bind inside their own network namespace, so chat is their only surface. Data-store collaboration was unaffected, because the store was already on the host and reached through a gate rather than a mount.

All the agents together sat at 1.4GB of 7.9GB RAM. Splitting a container is cheap. Believing that a 0700 directory is isolation is not. The rule we wrote down at the end of that day still stands: never mount one agent's directory into another's container, because that is the entire control.

The full post-mortem on that split, including the compose layout, is in one container per agent.

Where should an agent's data live if not inside the agent?

On the host, root-owned, served through a gate. The constraint that dictates this is simple: inside one container every profile can read every shared file, and terminal access defeats folder permissions anyway. So the store has to sit outside the agent's reach entirely and be handed to it one request at a time.

The shape we landed on is git-backed markdown in a root-owned directory on the host, read through an SSH forced command. Each profile has its own key, with its profile name baked into the forced command itself. An agent cannot claim to be another profile, because the profile name is not a parameter it supplies. The access control list is a root-owned YAML file of glob patterns, and it is evaluated by the reader, not by the caller.

Every file carries YAML frontmatter with a status and a visibility field. Lifecycle is a field change and a folder move, never a delete, so history stays in git. Compensation, HR events and identity or banking records are tiered to HR only; role and reporting line are team-visible.

Verified live rather than assumed: the COO agent is refused on compensation and on identity records, and allowed on the same person's profile record. A path-escape attempt using relative parent directories is refused outright.

The permission test that could not fail

A trailing comment after a YAML list broke the ACL parser and silently gave the founder profile zero permissions. Nothing errored. The reader simply refused everything.

We caught it only because the test suite asserted a positive case alongside the negatives. This is worth stating as a rule of its own: an access-control test made entirely of denials passes just as happily against a broken parser as against a working ACL. If your permission suite has no assertion that says this caller must get this file, it is not testing permissions. It is testing that your system is off.

If you are designing this layer for your own agents and want it reviewed by people who have already broken it once, that is what our AI agent development team does.

Why did we replace the file ACL with a Postgres control plane?

Because a file ACL is enforced by a program we wrote, and every endpoint that forgets a check becomes a leak. Row-level security moves the decision into the database, so an endpoint that forgets a WHERE clause returns nothing rather than everything. The API stopped deciding what a caller may see and started only stating who is asking.

Concretely: each request opens a transaction, sets the role and the caller identity for that transaction only, and runs the query. Every filtering decision is a policy the database evaluates. That is the inverse of the file layer, where the correctness of every read depended on the reader program getting the glob match right.

The superuser hole we created within the hour

Before writing a line of the API, a check on the application role returned that it was both a superuser and a bypassrls role. Every policy we had written the day before was inert on that connection.

Our row-level security tests had all passed, because each of them explicitly set a non-superuser role first. The tests were correct. The default path was wide open. That combination is the dangerous one, because a green test suite is exactly what stops you looking.

Worse, we had added FORCE ROW LEVEL SECURITY the previous day specifically believing it closed this gap. It does not. FORCE subjects the table owner to row security; it has no effect on a role carrying the bypass attribute. The PostgreSQL documentation on row security policies states it plainly: 'Superusers and roles with the BYPASSRLS attribute always bypass the row security system when accessing a table.'

This is the managed-platform service-role-key problem, which we had deliberately avoided by not using a hosted backend, recreated locally by accident inside an hour. The standard advice is that the service role key must never leave the server. That understates it. The danger is not where the key lives. It is that a bypass role exists at all and is the obvious thing to connect as, because the standard Postgres compose environment variable creates the cluster superuser and that is the credential sitting in front of you.

The fix was a second role: no superuser, no bypassrls, no inherit, owning no tables and named in no policy. The no-inherit part is not cosmetic. Without it the role silently holds the union of every profile's privileges the moment it connects.

We verified the failure mode rather than the success mode. Before any role is set, the API's connection gets permission denied for the people table, not an empty result. Fail-closed by construction, rather than fail-closed by everyone remembering to set a role.

Why the API answers 404 and never 403

Because a 403 tells the caller that the record exists. An employee querying a colleague's compensation should not learn there is a compensation record to query. Row-level security returns zero rows, the endpoint genuinely cannot distinguish not permitted from not there, and 404 is the honest answer as well as the safe one.

The matrix we verified over HTTP rather than in SQL:

• HR agent reads an identity record: 200.
• COO agent reads the same identity record: 404.
• Employee reads their own employment terms: 200.
• Employee reads a colleague's employment terms: 404.
• Bad key: 401.

Two of the most sensitive tables are withheld at the GRANT level as well as by policy, so a permissive policy added by mistake in six months still cannot leak them. The API translates the resulting insufficient-privilege error into the same 404 that an empty policy result gives, so the caller cannot tell withheld from absent.

Every call is audited, refusals included. The audit table shows one agent's denied identity read and an employee's denied attempt on a colleague's terms. Successful reads tell you the system works. Refused ones tell you what people and agents are reaching for, which is the more interesting log.

Why can our agents not reach the database at all?

Because only the API joined the agent network. The database did not. From inside an agent container, the API's health endpoint answers normally and the database hostname fails to resolve at all: temporary failure in name resolution. There is no credential, no route and no DNS entry.

This is the part of the architecture we would keep if we had to throw everything else away. A tool that lacks a connection string is a tool one edit away from having one, and these agents write code for a living. A container that has no route to the database cannot be talked into acquiring one, because the network topology is defined on the host and the container has no handle on it.

The distinction is worth stating precisely: the networking is the control, not the tool. Anything enforced inside the agent's process, its filesystem or its prompt is enforced by something the agent can rewrite. Anything enforced by a network namespace it does not control is not.

Verified per agent, through the real tools rather than a scripted test: JARVIS reads an identity record and gets it. Taurus reads a client's commercial record and gets it. Taurus reads the same identity record JARVIS just read and gets 'not found, or not visible to you'. Neo lists clients and gets the full set. Kiara reads employment terms and gets them.

The tool's own docstring states that it cannot widen access, and says why. No flag, no argument and no edit to the file changes what comes back, because the policy is in the database and the container cannot reach the database. That sentence is in the docstring deliberately, because the instinct on hitting a 404 is to go looking for a permissions flag, and the next engineer to hit one should find the explanation where they are already looking.

What is the write gate, and why is an in-container approval code theatre?

The write gate is a root-owned executable on the host, behind an SSH forced command, that holds the write credentials and issues a one-time approval code to the founder directly. The agent submits proposals and relays the typed code back. It never sees the code and never holds the credentials.

An approval code that lives inside the container is theatre. The agent can read it, or brute-force a short hash, and it holds a terminal with which to do either. The code and the write credentials have to live where the agent has no path to them, which in practice means the host, root-owned, reached only through a forced command.

The codes are single-use, bound to a specific action, expire in 60 minutes, and auto-reject with an alert after three wrong attempts. Execution is registry-only: ten registered action types covering host commands, container and compose restarts, accounting writes, ad-platform changes and CRM writes. Anything unregistered is refused at the door rather than evaluated.

We tested it adversarially rather than functionally. The agent cannot read the code through the status verb. Wrong codes are rate-limited. The credentials file is unreadable to it. The SSH key yields no arbitrary command. Unregistered action types are refused. The full submit, code, approve, execute, audit loop was proven end to end on a real action, not a mock.

The restriction we shipped and then reverted

At one point we made the agent's own tools and skills read-only. That was reverted within a day, on founder pushback, and the reasoning is worth keeping.

With the gate in place, freezing the skill files added almost nothing, because the gate is enforced server-side and editing a client-side wrapper gains the agent no privilege at all. What it did do was disable the learning curator, which is one of the headline capabilities the platform exists to provide. We had traded a real capability for the feeling of a control.

The gate was then widened rather than narrowed: arbitrary root command execution on the host became a registered action type, gated by the same code. Maximum capability, single control point. Restrict the consequences, not the capabilities. An over-restricted agent is a failed agent, and the version of this system that got locked down hardest was also the version that did the least work.

The gate's full design, including why we used one messaging bot rather than two, is in the write gate.

What did this architecture get wrong along the way?

Five things, all of which passed a test or a review at the time. Listing them is more useful than listing what worked, because the failures share a shape: each was a control that reported success while doing nothing.

• The interface fallacy. We concluded that chat-only access was safe because the dashboard lacked per-profile access control. The agent holds a terminal regardless of the interface, so the conclusion was backwards, and it delayed the container split by a day.
• FORCE ROW LEVEL SECURITY, added specifically to close the bypass gap it does not close. Reading the documentation after writing the code, rather than before, cost us a day of believing the policies were live.
• A backup dying silently. Four live agents write while the archiver reads, so tar exits 1 with 'file changed as we read it'. With stderr discarded and a fail-fast shell option set, the job died on any night an agent happened to write mid-archive, and cron threw the exit code away. It had succeeded once, purely because nothing wrote during that particular second. The preflight check we were proud of never ran, because the script was already dead by the time it would have. A fix verified once, on a quiet system, is not verified.
• A schema tidier than the data. Seeding failed on a boolean column because the underlying records carried three states: true, false and partial. Coercing to boolean would have erased the distinction between done and half done across two dozen client records. The column became text. A schema that is neater than reality destroys information quietly, and the seed error was the only reason we noticed.
• A hardcoded identifier map. We reported a set of employment documents as misfiled, and they were not. Folder IDs had been transposed in a list carried across steps. Resolving parent chains from the root proved the original filing correct, and the false urgent item was retracted. Resolve identifiers from the source on every run, and never carry an ID map between steps.

The habit that caught most of these was an adversarial review loop: have an independent model audit the deployment, then verify its claims yourself rather than accepting them. Two passes found real defects, including truncated pagination that was silently understating totals and a complete absence of timeouts on outbound HTTP calls. Both passes also made confident claims that were simply wrong about our deployment. The review is worth running. The verification of the review is the part that is actually load-bearing.

What does an agent mesh like this cost to run?

Two virtual servers and about 1.4GB of RAM for all the agents combined. There is no orchestrator, no service mesh and no managed platform in the path. The expensive part was never compute; it was the four or five days spent discovering that controls we had already shipped were inert.

The two-server split is not about capacity. The box holding finance and HR credentials must not be the box engineers have root on, because root reads every plaintext credential present and file permissions are meaningless against it. Profile isolation protects agents from each other. It never protects anything from a human with root, and pretending otherwise is how you end up with an org chart where the security model is 'we trust the team', unwritten.

We also declined to build the obvious application. At our headcount an in-house HR system would never repay its maintenance, payroll is close to the worst thing to build in-house given how often statutory rules change, and the failures we had actually observed were data-discipline failures rather than missing screens. A new form would have produced the same gaps behind nicer interfaces. The agent is the interface: a sentence describing what happened becomes records created, an exit recorded, documents filed and a change log written.

For the same reason we initially refused to install a database at all. At roughly 50 records, a DBMS is a service to run, secure, migrate and back up, and the per-profile ACL would have had to be rebuilt inside it. We derived a read-only SQLite index from the markdown instead, carrying only non-sensitive fields so it could never become a route around the file ACL, then retired it once the control plane became authoritative.

That index earned its keep in an unexpected way before it went. A file tree hides gaps; a table makes them countable. The first two queries we ran against it revealed that not a single person record had an access register, and that the department field was missing on most of them. Neither gap was visible while the data was only ever read one file at a time.

How would you build this yourself?

In this order, because each step closes the hole the previous one leaves open. The sequence matters more than any individual choice of tool.

• Give every agent its own container and its own UID, and mount only its own home. If two agents share a UID, you have one agent with two names.
• Put the data outside the agent and serve it through a forced command whose scope is baked into the key. The agent must not be able to declare who it is.
• Move the access decision into the database with row-level security, and check the connecting role's attributes before you trust a single policy. Assert the denied case on a connection with no role set.
• Remove the route. The API joins the agent network; the database does not. This is the step most designs skip, and it is the one that survives someone editing the tool.
• Gate the consequences, not the capabilities. Approval codes and write credentials go where the agent has no path to them, and the registry of permitted actions is enforced at the gate rather than in the prompt.
• Test the positive case. A permission suite made only of denials passes on a broken parser, and you will not find out until someone needs a file.

The primitive: a limit the agent can edit is not a limit

Every control in this build that survived contact has the same shape. The thing enforcing the rule sits somewhere the agent has no path to: another UID, another mount namespace, another network, another machine, a root-owned file. Every control that failed shared a trust domain with the agent it was meant to constrain.

A 0700 directory under a shared UID. An approval code inside the container. A read-only instruction in a system prompt. A row-level policy that a bypass role ignores. Each of those looked like a boundary in a design document and behaved like a preference in production.

Call it the unreachable control. If your agent can read, edit or impersonate the thing that limits it, you have written a preference. Prompts are preferences. Networks are boundaries. When you are deciding where to put a control, the only question that reliably predicts whether it will hold is whether the agent could get to it on a bad day.

We are still building this in the open, and the running system is documented in the Neogen AI OS case study, with the day-to-day operating detail in running a Hermes agent in production. If you are putting agents anywhere near real company data and want the boundary designed before the incident rather than after it, talk to us.

Frequently asked questions about AI agent architecture

Do you need Kubernetes to run an agent mesh?

No. Ours runs on plain Docker Compose across two virtual servers, with separate networks doing the isolation work. Kubernetes solves scheduling and scaling across many machines. An agent mesh at company scale is a handful of long-lived containers that never move, so the orchestration layer would add operational surface without closing a single one of the holes described above.

How many agents do you need before this architecture is worth it?

Two, if they have different data access. The moment one agent is supposed to see something another is not, a shared container makes that distinction fictional. The cost of splitting is a compose file and a second UID. The cost of not splitting is that your access control list is enforced against a caller who can borrow anyone's key.

Does this architecture stop prompt injection?

No, and nothing at the model layer does. What it changes is what a successful injection reaches. An injected instruction can still make the agent try to read a colleague's compensation record; it gets a 404, because the decision was made by a database the agent has no route to. Assume the model will be convinced, and design for what happens next.

Does row-level security slow queries down?

Not measurably at our scale, which is thousands of rows rather than millions. RLS appends a predicate to every query, so the cost is the cost of that predicate: index the columns your policies filter on and it stays cheap. If you are pushing serious volume, benchmark it, but do not trade a correctness guarantee for a performance concern you have not measured.

What happens when an agent legitimately needs data it cannot see?

It gets a 404 and proposes a gated action instead, which a human approves with a one-time code. That path is deliberately slower than reading. Most requests that hit a wall turn out not to need the data at all, and the ones that do are exactly the requests worth a human glance before they execute.

Why keep markdown files if you have a database?

Because the markdown store is the narrative record and git gives it history for free, while the database is the authoritative source for anything queried across records. Agents read files for context and query the control plane for facts. The index we derived from the files was retired once the control plane could answer the same questions, but the files themselves stayed.