We Gave Four Autonomous AI Agents Real Jobs. Here Is the Complete Build Log.

Autonomous AI agents are systems that hold a defined scope of work, read live business data, do the recurring work of that scope on a schedule, and report back without being asked each time. We have run four of them on real seats at our agency since 24 July 2026: founder, HR, operations and revenue. This is the build log.

Five weeks took us from an empty container to four agents holding 111 scoped skills, a governance gate that lives outside every agent, and a control plane the agents cannot connect to at all. It works. It also spent six days completely dead while every health check reported green.

We kept the log from day one because the failures are the useful part. Two of them have already become their own posts: the runtime failure modes of a Hermes agent in production and the six-day silent outage. This is the chronology those two hang off. If you want the finished system rather than the story of building it, that is the Neogen AI OS case study.

What are autonomous AI agents, and how are they different from a chatbot?

A chatbot answers when spoken to and holds no scope. An autonomous AI agent holds a seat: it has its own credentials, its own permitted data, its own recurring jobs, and its own escalation rules. The difference that matters in practice is not intelligence. It is permissions.

The market is arriving at this faster than it is succeeding at it. Gartner predicts that over 40% of agentic AI projects will be cancelled by the end of 2027, citing cost, unclear value and inadequate risk controls. Anushree Verma, Senior Director Analyst at Gartner, put the reason plainly: "Most agentic AI projects right now are early stage experiments or proof of concepts that are mostly driven by hype and are often misapplied."

The failure rate on the generation before this one was worse. MIT's Project NANDA reviewed more than 300 disclosed AI deployments alongside 52 executive interviews and found that 95% of generative AI pilots produced no measurable P&L impact, with deployment quality rather than model quality as the cause. Both numbers describe the same gap: the pilot works, the production system does not, and nobody wrote down why.

Why did we build autonomous AI agents for our own agency before selling them?

Because a system nobody has run with real consequences is a demo. We wanted the outages and the bad architectural calls to happen to us, on our own payroll data, before any of it reached a client. That also changed what got written down. When it is your own company, an entry saying "I filed a document containing a colleague's salary into a folder that colleague could read, minutes after writing the rule against exactly that" survives editing.

What did we decide before provisioning anything?

Everything that would be expensive to reverse. Four documents were written before the first container existed: the architecture of record, the build spec, the founder agent's operating persona, and its reporting contract. Roughly two days of writing bought us five weeks of not re-litigating decisions at 11pm.

Three rulings from that phase did the most work:

• Build order: apex seat first, then the seat whose data must be isolated from everyone, then the best-documented operational seat, then the least-documented one. We started with the founder agent because it is the only seat allowed to see across all the others.
• Reads are pre-approved, writes are gated. The agent may read anything in its scope and may dispatch work to other agents on its own judgement. Anything outward facing - client contact, money, publishing - stops for a human.
• Interrupts are preset, not judged. A fixed list of triggers earns an immediate message. Everything else waits for the scheduled digest. An agent deciding for itself what counts as urgent is an agent that will interrupt you about nothing.

The discovery interview that produced the founder agent took six questions and about three minutes, and it was incomplete. Our founder caught the gap himself: no question had asked what applications he actually opens each day. That inventory is now a required module, because the target for an agent's tooling is parity with the surface its human already works through. Missing it produces an agent that is technically capable and practically useless.

Why did the first container refuse to stay running for two days?

Because we were debugging the wrong layer. Every attempt to complete an interactive login died within seconds. We tried backgrounding it, tmux, setsid, a nested SSH session, and the vendor's own setup wizard. Eight one-time device codes expired unused before we looked at the container itself.

The container had restarted 342 times. On that image the supervision service is a no-op and the container's command is the real program, and the default command is an interactive chat interface that exits immediately with no terminal attached. Every restart killed every process we had started inside it. Our earlier stability check had run inside one of the up-windows and reported healthy.

The fix was one line of configuration, running the gateway in the foreground as the image's own documentation recommends for containers. The lesson cost two days: verify that the container is stable before you debug anything running inside it. We wrote the misdiagnosis into the log next to the correction rather than quietly replacing it, which is why we can still tell you what we believed at the time.

Interactive logins on a crash-prone container now start by pinning the entrypoint to sleep, so nothing can restart underneath them. The full set of Hermes runtime failure modes has the rest of these.

How do you connect a real business stack to an agent without leaking it?

One credential at a time, each one probed with a harmless read before it is wired to anything. We consolidated 38 keys into a single staging file, annotated each with the seat it was destined for, and moved values programmatically so none of them passed through a chat window.

Probing before wiring found three faults that would otherwise have surfaced as an agent confidently reporting nothing:

• One infrastructure token was dead. It had been rotated and the working copy lived in a different config file entirely.
• One analytics token returned 403 and looked revoked. It was not. The 403 came from the admin API; the reporting API it would actually be used against worked perfectly. Test the endpoint you will call, not the one that is convenient to call.
• Our CRM's edge firewall silently rejected Python's default user agent while accepting the identical request from curl. Setting an explicit user agent header fixed it. Google's APIs do not do this, so nothing in earlier testing had predicted it.

The catch that mattered most came after all of them were wired and tested. Every tool passed when we ran it by hand. None of them would have passed when the agent ran it, because the runtime loads a profile's environment file for its own use and does not export those variables into tool subprocesses. Our tests had been passing an environment explicitly and hiding the bug.

Every data tool now parses its own environment file on startup, and every tool is tested with a deliberately empty environment. A test that passes because of how you ran it is not a test.

What changed the day the agents got real numbers?

The agent stopped being a thing we demonstrated and became a thing that told us something we did not know. Search data went in first, then analytics, then the CRM, then the books. Once all four reporting sections had live sources behind them, we put the digest on a schedule twice a day and the agent became proactive rather than reactive.

The first scheduled run completed in about 31 seconds. It led with cash rather than with traffic, which is the order the reporting contract specifies, flagged that every open receivable was past due, questioned a stale supplier bill dated over a year earlier, and described the sales pipeline as "ornamental" because every opportunity in the CRM had been created without a monetary value on it.

That last one is the point of the exercise. Nobody had asked it to audit CRM hygiene. It could not report pipeline value, said so instead of estimating, and named the reason. That is a hard rule in its operating instructions: no figure that did not come from a real query, every figure names its source, a failed lookup is reported as a failed lookup. Confident fabrication is the failure mode that makes most business AI unusable, and it has to be designed against on purpose.

Which question changed the entire architecture?

"My developers have terminal access. Can they see everything?" The answer was yes, and it moved the whole deployment onto different hardware within a day.

Root on a machine reads every plaintext credential on that machine. File permissions are not a defence against root. Profile isolation protects agents from each other and does nothing at all against a human with a shell. Those are two separate walls, and we had built one of them.

So the rule we deploy under now is that the machine holding finance and HR credentials is not the machine engineers have root on.

One thing worth saying plainly: deleting the old data did not undo the exposure. Only rotation does that. Our founder chose not to rotate, and that is recorded in the log as an accepted risk with his name against it, not as a task nobody got around to. The distinction matters when you read the log back in six months.

What is a write gate, and why is a prompt not one?

A write gate is a mechanism outside the agent that authorises consequential actions. It is the core governance primitive of the whole system, and the reasoning that produced it is the single most portable thing in this log.

Two ideas that sound like controls are not:

• "Read-only by default" enforced by prompting is a convention. An agent processing a hostile document cannot distinguish its owner's instruction from text injected into the thing it is reading. The instruction and the attack arrive through the same channel.
• An approval code stored inside the agent's container is theatre. The agent can read the file. A three-digit code can be brute-forced by the thing being gated. The code and the write credentials have to live somewhere the agent cannot reach, or they are decoration.

What we built instead: a root-owned executable on the host behind a restricted SSH command, with the write credentials in a file only root can read. The agent submits a proposal. A single-use, action-bound code with an expiry goes directly to the human. Three wrong attempts auto-reject and alert. Execution is registry-only, so an action type nobody registered is refused at the door rather than evaluated.

We attacked it with our own agent rather than reasoning about whether it held: it could not read the code through the status command, could not read the credentials, could not obtain a shell from the key, and could not execute an unregistered action type.

Then we loosened it, on our founder's instruction, which is the correction worth copying. We had made the agent's own skills and tools read-only; he reverted that. With a server-side gate in place, freezing local files added almost nothing while disabling a headline capability. Restrict the consequences, not the capabilities. Version-control the drift instead of forbidding it. An over-restricted agent is a failed agent.

If you want this shape built against your own systems rather than assembled from a build log, that is what our AI automation service does, and AI agent development is the smaller version of it: one agent, one system, one seat.

Where should company data live if the agent can read every file it can reach?

Outside the agent, served through a gate. This is the write gate lesson applied a second time, and we only reached it because a question forced it: how do you keep everything in one place when not everyone should see everything in it?

Inside a single container every profile can read every file, and terminal access defeats folder permissions. So the store sits in a root-owned directory on the host that no container mounts. Each agent gets its own key, with its own identity baked into the restricted command that key is locked to. An agent cannot claim to be another agent, because the claim is not a parameter it supplies.

Access is a root-owned rules file of path patterns. Sensitivity is tiered by audience rather than by topic: role and reporting line are visible to the team, while compensation, HR events and identity documents are restricted to one seat. Every document carries structured frontmatter declaring its own status and visibility, and the whole store is in git, so a person leaving is a field change and a folder move rather than a delete. History stays.

We verified the tiering by exercising it rather than reading it: the operations agent is refused on compensation and identity documents, allowed on the role profile, and a path traversal attempt is refused. Rules files had passed review while silently granting the wrong thing, so reading them is not evidence.

The bug that only a positive test could have caught

A trailing comment after a list in the rules file broke the parser and silently gave the founder profile zero permissions. Every deny test still passed, because everything was denied. It surfaced only because the test suite also asserted that permitted reads succeed. A permissions test made entirely of negative cases will report perfect health on a completely broken system.

Do autonomous AI agents need a database, or is markdown enough?

At roughly fifty records, no. A database server is a service to run, secure, migrate and back up, and the per-agent access rules would have to be rebuilt inside it. Markdown is the format the agents read natively and git gives you history for free.

Querying was genuinely weak though, so we kept markdown as the source of truth and derived a read-only index from it, regenerated when files change. Agents read files for narrative and query the index for aggregates. The index is a deliberate boundary as well as a convenience: it carries only non-sensitive fields, so it can never become a route around the file permissions.

The unexpected return was that the index made gaps countable. The first two queries we ran revealed that not one person on record had an access register listing the systems they had been granted, and that a department field was missing from most records. A folder tree hides that. A table with a null column in it does not.

That gap was not cosmetic. Someone had already left without notice, and nobody could say what access that person still held. We did not have the register until a query counted its absence.

What broke when one agent became four?

Two things, both invisible until something exercised them, and both of them corrections to entries already in this log.

Three separate bots, one conversation

Our founder, HR and operations agents all introduced themselves as the HR agent. Asking the founder agent about the CRM produced "that belongs to another agent", because the founder profile was not the thing answering.

The gateway namespaces session keys per agent, but the fallback branch of its lookup matched on user and chat identifiers only, omitting the agent. In a direct message those two identifiers are the same value, so one person allowlisted on three bots collapsed onto a single session belonging to whichever agent got there first.

Blast radius was nil, because only our founder was in more than one allowlist and the bleed ran downward onto the seat authorised for all of it. The latent risk was the finding: any team member added to a second bot's allowlist would have inherited that agent's credentials and data identity. There is no configuration option for this. We reported it upstream and adopted a hard rule instead: one messaging account appears in exactly one agent's allowlist, no matter how convenient the exception looks.

One container, one user, every credential

All three agents ran in one container as the same system user, with terminal and code execution enabled. Directory permissions isolated nothing, because one user owned all of them. Any agent could read the founder agent's environment file and its entire secrets directory.

Describing that as "one agent can read another agent's files" undersells it. Holding the founder's data key inside the operations agent's trust domain, the host-side access check answers as the founder. The permission system was working correctly and completely. It simply was not us asking.

This also corrected an earlier entry of our own, which had concluded that team members could safely be given chat access because credential isolation between profiles was fail-closed. That clause was wrong, and being wrong in writing is why it sat unnoticed for a week. The agent holds a terminal regardless of which interface you talk to it through, so restricting the interface cannot contain a shared trust domain.

The fix is the rule we would now apply on day one: one container and one system user per agent, with no agent's storage mounted into another's. That gives two independent layers, since the sibling's path does not exist in the mount namespace and the user could not read it if it did. All four agents together use under two gigabytes of memory, so there is no cost argument for sharing.

What do the four autonomous AI agents actually do?

Four seats, four scopes, 111 scoped skills between them, and a different refusal list for each. They read live data, run recurring jobs on a schedule, produce digests, and escalate on defined triggers rather than on a judgement they are not qualified to make.

• JARVIS, the founder seat. The only agent allowed to see across all the others. Twice-daily briefs covering cash, pipeline, client health and delivery, each section from a live source. Holds the broadest credential set and the books.
• Kiara, the HR seat. Holds compensation, identity documents and the statutory picture, and is the only seat that does. Its most useful skill encodes employment thresholds and is required to say when a policy document is stale rather than quote it, because we crossed the ten-employee line that changes several obligations while our written policies still described a smaller company.
• Taurus, the operations seat. Received most of the founder's skill library, minus finance, minus HR, and minus full document-storage access, since granting that would have handed over HR data through the back door while the front door was still locked. Refused on compensation and identity, allowed on receivables, and holds zero finance credentials.
• Neo, the revenue seat. Built last and least documented, so it was the seat that tested whether the assembly line worked without a thick manual behind it.

Kiara's counting rule shows what a scoped agent is for. A headcount is not an employee count: some people are not on employment contracts, some are interns, and statutory insurance counts only those under a wage ceiling. The skill has to show its working rather than quote a number, and is forbidden from asserting the company is compliant, because a threshold being crossed is not a registration existing. It can see the headcount. It cannot see whether the filing was made.

How did we get the agents speaking Malayalam without paying for it?

By being wrong for a day and then being corrected by a human with ground truth. All four agents now take voice notes and reply in speech, in English and in Malayalam, at zero ongoing cost. The path there is the best example in the log of a misdiagnosis that nearly bought a product.

Our conclusion after a morning of testing was that the speech model could not do Malayalam. That was wrong. It heard Malayalam correctly the entire time and failed only at choosing which script to write the output in. We were one purchase order away from buying a paid regional speech vendor to solve a problem that a single prompt parameter fixed. What overturned it was not more testing. It was our founder sending three sentences of romanised Malayalam and saying that this, not the native script, was the output he wanted. We had been benchmarking against the wrong target.

One technique from that week is worth stealing:

• Grade synthesised speech by transcribing it back through your own speech-to-text and comparing the strings. "Does this sound right?" becomes an objective check that needs no listener. It is what caught an agent speaking gibberish, after we had set a Malayalam voice for all text without testing whether that voice can read Latin characters. It cannot.

Two configuration lessons came out of the same week, and they are the same lesson twice. A model setting was accepted and silently ignored, so production ran a different model for hours while the log claimed otherwise. The gibberish happened because the software accepted a voice setting that was invalid for the content. A configuration knob existing is not evidence that anything reads it.

What did the governance layer actually catch?

Three data exposures, three silent zeros, and one hole we created ourselves an hour after declining a product specifically to avoid that hole. All of it came from a single session spent turning a folder of documents into a governed store with 117 documents tiered by audience and 110 enforcement assertions.

Three exposures, all the same shape

Content sitting in a folder more permissive than the content deserved, with the access rules working perfectly in all three cases. A strategy document filed by topic without being read, containing a colleague's salary and a negative assessment of that colleague, readable by her. A pending-items file that declared itself restricted while sitting where two other seats could read it. A duplicate of a hiring policy leaking interview thresholds to everyone.

And after splitting the second file, we left a backup of the original in the same folder, defeating our own fix. Splitting a file achieves nothing if a copy of the original stays readable.

Three silent zeros, each printing a success line

A file handle position read during iteration, a stray newline in a generated file, and a frontmatter split that matched a comment containing the same delimiter. One reported zero compensation across every person. One left a staleness check permanently red, which is the same as muted. One reported zero recurring revenue across every client.

Every one of them exited successfully. None raised an error. Each was caught only because a human compared the number to an expectation instead of accepting it, and the same shape recurred later in the same week when a backup script began exiting silently on any night the agents happened to be writing during the archive.

The hole we recreated within the hour

We declined a hosted database for the control plane, self-hosted Postgres instead, and wrote row-level security policies. Then we connected as the default superuser, which carries a bypass on those policies, so every policy written the hour before was inert for the obvious connection. Our tests had passed only because they explicitly assumed a restricted role first.

Forcing row-level security does not close that. It binds the table owner, not a bypass role, which is exactly what we had believed it did. We had reproduced the precise hole we declined a product to avoid, in the same session, in the same file.

The through-line across every one of these is one sentence. The query layer said read-only. The pending file said restricted. The rules file said deny. The speech config said one model. The org chart said it was current. Every one of those was believed until something exercised it, and every one was false. Test the behaviour, never the declaration.

What did six days of silence cost us?

Every subordinate agent stopped working for six days and nothing reported a fault, because nothing failed. A global model identifier changed, the runtime refuses to run a job pinned to a model that no longer exists, and every unpinned scheduled job simply did not start. No errors, because no executions.

Every control was working. The monitoring watched for failures and there were none. We found it only because our founder forwarded an unrelated escalation, underneath which sat an agent that had verified every credential thoroughly before reporting it could not reach a machine it does not run on. Correct diagnosis, wrong premise.

The full teardown, including how we dated an outage nobody noticed, is in the AI agent observability post. The finding worth repeating here is the one that came out of the same audit, because it is about governance rather than monitoring.

86% of our approvals were not decisions

Of 138 gated approvals in the fortnight to 8 August, 119 were requests to run a command on the host. Only 19 were registered business actions. Our founder approved 106 and rejected 4, which is a 96% approval rate. That is not a filter, it is latency. And 42 of the 106 he approved then failed anyway.

The cause was not a permissive gate. An agent wanting to run its own script had no lane except asking for root on the host, so routine work escalated to the most dangerous verb in the system. If the only write lane is root, every routine task becomes a root request. That is privilege escalation by absence of a lower lane, and the fix is not to tighten the gate but to build the lane that should have existed: run the script inside the agent's own container as its own user, which is strictly less privilege than what it replaced.

Gating the wrong thing is worse than not gating, because it trains the person holding the codes to approve without reading. We also found the prompt itself was lying: approval messages were being sent with markdown formatting applied to unescaped content, so a script filename containing underscores rendered with the underscores stripped. The name shown for approval was not the name that would run. Never apply formatting to text a human is being asked to authorise.

More of these, including the ones with no tidy lesson attached, are collected in twelve ways our AI agents failed.

What would we do differently on day one?

Six things, in the order we would now do them:

• One container and one system user per agent, from the first agent. We arrived here after discovering that a shared trust domain makes every other isolation layer decorative, and retrofitting it cost a day.
• Build the low-privilege execution lane before the approval gate, not after. Otherwise the gate fills with permission bugs wearing a governance costume and stops being read.
• Write the monitoring to detect absence, not just failure. Ours watched for errors and was blind to six days of nothing happening.
• Pin the model identifier per job. A global default that moves silently invalidates every job that trusted it.
• Assert positive cases in the permissions test suite. A suite of pure deny tests passes perfectly against a system that denies everything, including to the people who should have access.
• Keep the build log as you go, and record corrections next to the claims rather than instead of them. Several of our entries are wrong and each is followed by the entry that disproves it. Commit messages will not do this for you, because what the log carries is the reasoning and a subject line loses it.

Frequently asked questions

How much does it cost to run four autonomous AI agents?

The infrastructure is one virtual machine and all four agents together use under two gigabytes of memory, so hosting is not the meaningful cost. Model inference is, and it scales with how much scheduled work you give them rather than with headcount. Our speech layer, both directions and both languages, runs on free tiers at no ongoing cost at all.

How long did this take to build?

Five weeks from an empty container to four agents in daily use, working alongside client delivery rather than instead of it. The first agent took three days to reach live data. The remaining three were faster because the assembly line existed by then, which is the argument for building the apex seat first and treating it as the template.

Can a company that does not have engineers do this?

Not from scratch. Several of the decisions in this log are security decisions with consequences that are invisible until exercised, and the failure mode is silent rather than loud. What a company without engineers can do is run the result, because the daily interface is a chat window and a scheduled brief. Someone has to own the governance layer, and that person does not have to be the person who built it.

What happens when the model provider changes something?

It breaks, and you find out only if you are watching for absence. A model identifier changing under us is what killed six days of scheduled work. Pin the model per job, monitor for jobs that did not run rather than jobs that failed, and treat every provider-side default as something that will move without telling you.

The primitive: enforce it outside the thing being enforced

If one line survives from this build, it is that one. Every control in the system that held under testing was enforced somewhere the agent could not reach, and every control that failed was enforced somewhere it could.

The approval code lives on the host, so the agent relays it without ever seeing it. The data access rules live in a root-owned file outside every container, so no agent can widen its own scope. The agent's identity is baked into a restricted command rather than supplied as a parameter, so it cannot claim to be another agent. The database policies are enforced by the database and the agents cannot connect to it at all. Each of those held when we attacked it.

What failed was every control that lived inside the thing it was meant to constrain: a prompt saying read-only, a visibility field in a file's own header, an approval code in the agent's container, a permission model resting on directory ownership under one shared user. An agent can be argued with. A host-level rule cannot.

The companion rule came from our worst week. A stated control is not a control until you have exercised it and watched it refuse. Everything in this log that was true on paper and false in production was true on paper for weeks first.

We are still building. The log is still open, and the failures still go in it. If you want an agent mesh built against your own systems with the governance layer in place before the agents are, talk to us.