Stated Controls Are Not Controls

An AI governance control is only a control if the system refuses the action. A restriction written into a prompt, a config file or a policy document is a description of enforcement, not enforcement. In our own agent mesh we found declared restrictions the runtime never applied, and granted permissions the prose withheld. Test the behaviour, never the declaration.

We have been building an internal AI operating system since July 2026: four agents on isolated profiles, a write gate, a row-level-security control plane and an access-controlled data store. The most repeated failure across that build has nothing to do with model quality. Somebody wrote down a restriction, everybody read it, and nobody ran the action to see what happened. Here it is named, with the receipts from our AI OS build.

What is an AI governance control, and when does it stop being one?

A control is a code path that refuses. A declaration is a sentence saying a code path refuses. In a document the two look identical; in production they are unrelated. This is why NIST's AI Risk Management Framework keeps Measure and Govern as separate core functions. Govern is where you write the policy. Measure is where you find out whether the system does it.

The security field settled this argument twenty-five years ago. Bruce Schneier, writing in Information Security in April 2000: "Security is a process, not a product." A policy file is a product. Exercising it against the running system is the process, and it is the part teams skip because the document already reads correct.

OWASP's 2025 list for LLM applications ranks Excessive Agency at LLM06, attributing it to excessive functionality, permissions or autonomy. That is the right framing for the outward risk. The inverse showed up more often in our mesh: agents behaving far more timidly than their permissions allowed, because the prose they read on every turn disagreed with the gate that actually decided.

Why was our COO agent asking permission for decisions it already owned?

Because its instruction file told it to, and that file was out of date. The approval gate had already granted the agent every write action its own rule was sending to the founder. The permission was granted in the runtime and withheld in the text, and the text is what the model reads every time it reasons.

On 9 August, seventeen messages passed between Taurus (our COO agent) and JARVIS (the founder agent) in three hours over a naming standard and a set of record IDs. Five of JARVIS's replies came back as deliberate policy refusals. The thread ended as a decision memo on the founder's phone asking him to approve a naming convention. Taurus held his own API key for that platform throughout and could have executed all of it unilaterally.

The tell was cost, not error. Nothing broke, no alert fired, no data leaked. The system spent three hours of two agents' attention and a founder's patience enforcing a rule that no longer existed. That is the signature of this failure: invisible to every monitor you own, because from the runtime's point of view everything worked.

The sweep afterwards found the same defect in both other subordinate agents. One had nine self-service action types granted in the gate and denied in her prose. Our revenue agent's file said the CRM was read-only, while contact and opportunity updates had been in his self-service set for as long as that set existed. Thirty-three of his skill files opened by telling him he was a different agent, copied across without changing the header, against a file that forbade exactly that. A hundred and twenty-nine skill files across four agents still named an approval route we had replaced. Full write-up in agent authority lives in its prose.

When one agent behaves more timidly than its permissions allow, the others are not fine. They are quieter about it.

What happens when a config setting is accepted and silently ignored?

It persists, it validates, it shows up in every status readout, and it does nothing. We set our speech-to-text model to a specific large Whisper build and logged the change. Production ran a different, distilled model for hours while our own build log stated otherwise.

One branch of the provider function read a module-level default instead of the config section. Every other provider in that same function read its config properly; one did not. The documented environment-variable override was inert too, resolved at import time by a call that never sees the profile's environment file. We added it, verified it did nothing, and removed it.

The same shape sat in our access layer. Three of the five deny patterns on every employee profile were decorative, because the function short-circuited on a match against the requester's own record and returned true before the deny list was consulted. The config declared a restriction the runtime never had the chance to apply. A config knob existing is not evidence it is read.

Why did a passing test suite prove nothing?

Because it tested the wrong layer. Our access-matrix suite was green at 110 of 110 assertions and we claimed in writing that it stopped the day's failures repeating. It would have caught none of them. In all three exposures the access layer worked perfectly; the fault was content sitting in a folder more permissive than the content deserved.

A conformance test proves the enforcement layer matches its own config. It says nothing about whether the config expresses the policy you meant. Two adjacent failures came from over-trusting a green result:

• A keyword scan for staff names and money terms flagged one section only because a colleague's first name appeared in it. The real exposure there was a negative assessment of the whole team, matching no keyword on the list. A clean scan is evidence the scanner did not match, not that the content is safe.
• A privacy test reported "salary figures on page: 0" and was counted as a pass. The page was returning a 500. Every check now prints the status code beside the finding, because a missing grant raises a privilege error where a policy returns zero rows, and a checker that only counts matches cannot tell those apart.

The first version of our content scanner returned fifteen findings of which two were real. A scanner at thirteen percent precision gets muted within a week, which is worse than having none, because now you believe you are covered.

Where this applies to a client build

Every agent we ship has a permission surface, and the gap between what its instructions claim and what its credentials allow is where the incident comes from. If you are scoping an agent build, our AI automation services page covers how we structure the gate, the credential boundary and the verification pass before anything touches production data.

How do you test behaviour instead of declaration?

Run the action as the constrained identity and record what the system returns. Not a config review, not a read of the prompt: the call, made under the profile in question, with the response captured. Each practice below came out of a case where the document and the behaviour disagreed.

• Exercise the permission as the profile, never as yourself. A document carrying an HR-only marker in its own header was handed to two leadership roles by the access file anyway. We found it by running a row count under one of those roles and getting thirteen rows back. The document and the ACL had contradicted each other for weeks and the ACL was winning.
• Write the assertion before the fix. When the founder ruled on three access questions we encoded the ruling in the suite first. It failed on exactly two assertions and nothing else, which named what had to change, then proved the fix addressed it and proved nothing else moved.
• Test the regression, not just the new denial. A permissions change that verifies only the newly-denied path is half a test.
• Restrict the agent, not only the person. Our agents share the roles of the humans they support, and a restriction applied to a person but not to their agent is not a restriction.
• Let verification decay by itself. A confirmation is stored against the document's hash at that moment; edit the document and the state reverts to needs-confirmation. Without that, "verified" only ever means "was verified once", and a record can be edited into anything while still showing green.
• Make the process that writes unable to approve. We proved ours as behaviour rather than reading it off the config: a submitter approving their own change, a leader approving above their risk tier, the founder approving his own proposal and direct database inserts bypassing the app all returned refusals in one recorded run. Described in full in the write gate.

What does skipping this cost?

Mostly the ability to tell a working system from a broken one. The morning we switched on a verification layer over our operational records it found 374 facts waiting on a human: 15 with no source, 30 marked TBD, 301 never confirmed by anyone, 28 with a recorded gap. Zero verified, because verification had not existed until then.

Nothing had been broken the day before. It had been unmeasured, which looks identical from outside and is the state our agent mesh sat in for six days while every dashboard stayed green. A scheduled poller there had a repeat interval of 5,256,000 minutes and a next run date in 2036, while its display string still read "every 2m" and its counter showed 866 completed runs. It had fired 866 times and would never fire again.

The primitive, and the one line worth keeping: a stated control is not a control. Our query layer said SELECT-only, a file header said HR-only, the access config said deny, the model setting named one build, the org chart said it was current. Every one was believed until exercised, and every one was wrong.

Frequently asked questions

Is this just an argument for writing better system prompts?

No, it is an argument for not relying on them for enforcement. Prompts are the right place to describe intent, tone and scope. They are the wrong place to hold a security boundary, because a prompt-injected agent cannot distinguish your instruction from injected text. Anything that must not happen should be impossible at the credential or gate layer, with the prompt agreeing.

Does an approval gate solve this on its own?

It solves the outward half. A gate stops an agent doing what it should not. It does nothing about the failure we hit more often, an agent refusing work it owns because a stale sentence told it to escalate. Gates need a matching audit in the opposite direction: for every capability the gate grants, check that nothing in the agent's instructions denies it.

We are a small team with no test suite. Where do we start?

Take the ten most consequential things your system claims it will not do and try each one by hand, under a real constrained account, recording the response. That afternoon will tell you more than any policy review and needs no infrastructure. Automate only the assertions that failed, because those are the ones that will drift again.

How often should controls be re-exercised?

On every change to the layer that enforces them, plus a schedule slow enough that you actually run it. Ours runs the full matrix on each access change. The failure to design against is not an infrequent check but a check that quietly stops running while its status display keeps reporting healthy.

Does any of this apply if our agents are read-only?

Especially then, because "read-only" is usually the declaration under test. One integration key of ours was described as read-only in three separate documents. It exposed dozens of write actions, including create and update. The agent holding it was reporting itself unable to see records it could have created.

If you run agents against live systems and have never watched one of your controls refuse an action, you do not have a control yet. Talk to us and we will walk your setup with you, starting with the ten things it claims it cannot do.