Skip to content

Insights

  • ai
  • agents
  • architecture
  • governance
  • evals
  • federal

Anatomy of a Governed AI Factory

The Colophon case study made the claim and the routing essay made the model calls; this is the architecture that makes a governed AI factory real and auditable, from the coordination plane to the eval loop.

Date

~9 min read

The case study made a claim: a governed AI factory can emit its authorization evidence as a byproduct of shipping, not reconstruct it in a scramble at the end. The routing essay explained how the model calls get made: boundary first, failure mode second, disagreement as the eval that counts. Both are true. Both are also the kind of claim a serious evaluator is right to distrust until they can see the machine underneath.

So this is the machine. Not the source, not the prompts, not the thresholds, but the shape of the system: how work coordinates, where it is contained, which gates it clears, and how the whole thing is measured. One sentence carries the whole design, the same one the case study closes on: humans decide, the bench ships. What follows is that sentence rendered as architecture.

The test I hold it to is simple. If I handed this description to an ISSO who had never seen the platform, could they trace how a change moves from an agent’s first keystroke to a production system, and name the control at every step. If they can, the architecture is doing its job. If they cannot, the “governed” in governed factory is marketing.

The coordination plane

A bench of sixty-plus specialist agents is not a factory. It is a crowd. What makes it a factory is Stationarius, the coordination plane that decides who does what, in what order, and who has to sign off.

Agents do not talk to each other in free text. They talk through Colloquy, a typed wire protocol. Every message is a structured, validated object with a known schema, an origin, and a place in the record. That single choice removes an entire class of failure: an agent cannot smuggle an instruction, an artifact, or a decision through an unlabeled channel, because there is no unlabeled channel. Everything that moves is typed, and everything typed can be checked.

Work escalates through three tiers. Peer agents resolve what they can among themselves. What peers cannot settle rises to arbiter agents, whose job is to adjudicate rather than to build. And what an arbiter should not decide alone, anything production-impacting, cross-cutting, or touching compliance, rises to the Change Control Board, whose seats are held by people. Escalation is the default path for uncertainty, not an exception you have to remember to invoke. The system is built to move a hard call up, not to let a confident agent resolve it quietly.

Underneath all of it sits Decretum, an append-only store of decisions and precedent. Every consequential call (who made it, on what, and why) lands there and stays. Decretum is not a log you can rewrite. It is the memory the factory reasons from, so that a decision made once becomes precedent the next agent has to reckon with.

State, isolation, and the rule an agent cannot break

We do not turn an agent loose as a general intelligence roaming the codebase. We scope it to least privilege by default: it sees the inputs its task requires and no more, it runs in an isolated context, and its capabilities are granted for the job in front of it rather than held ambiently. This is the same discipline we apply to human operators in an accredited environment, expressed for non-human actors.

We isolate more than what an agent can see. We isolate what it can do. An agent operates inside a sandbox that is torn down when its task ends, so state does not leak from one unit of work into the next, and one agent’s mistake does not silently become another’s starting assumption. Two agents working the same area do not share a mutable scratch space they can each corrupt. They coordinate through Colloquy and settle differences through escalation, which is the whole reason the coordination plane exists. Shared state is a channel, and every channel in this system is typed and recorded.

We enforce the boundary rule from the routing essay here as structure, not policy. CUI-bearing inputs never cross into a commercial endpoint, and we hold that line with static analysis and network egress control, not with an agent’s good judgment. An agent cannot choose to route sensitive data outward, because the path is not reachable from where it runs. Sovereignty is a property of the topology, not a checkbox the agent is trusted to honor.

The rule that ties isolation to governance is the one that matters most: an agent cannot reach around a gate. There is no side door where a change reaches production without passing review, because merge and deploy are not capabilities an agent holds. The only path forward is through the gate chain, and the gate chain answers to people. An agent that wanted to skip review would find nothing to skip to.

The gate chain

Every increment an agent authors enters the same chain, and we build that chain to be adversarial. Five review lenses look at it, each with a different question. Code review asks whether it is correct and maintainable. An adversarial reviewer tries to break it. Test-quality review asks whether the tests would actually catch a regression. Static application security testing hunts the vulnerability. Continuous-integration parity confirms it behaves the same in the pipeline as it did in development. Anything user-facing adds an accessibility lens. A change that cannot survive its own reviewers does not advance.

When a change is production-impacting, cross-cutting, or touches the compliance domain, it does not merge on the strength of clean reviews alone. It goes to the human Change Control Board. The board is where the judgment agents are not entitled to make gets made: what ships, at what risk, and who accepts it. The reviews inform that decision. They do not replace it.

The part that makes this a governed factory rather than merely a fast one is what falls out of the chain as it runs. The Risk Management Framework artifacts (the system security plan, the POA&M, the STIG checklist, the evidence bundle) are emitted as a byproduct of the same gated process that produces the code. Nobody reconstructs them later. The gate that approves a change is the same event that records why it was safe to approve, and that record is the evidence. This is the inversion the case study described, seen from the inside.

Provenance, rollback, and failure containment

We judge a governed system less by how it behaves when everything works than by what happens when something does not. So here is the failure story.

Provenance is total because Decretum is append-only. Every change carries an unbroken chain from the decision that authorized it to the artifact that shipped, and that chain is queryable after the fact. When someone asks why a thing is the way it is, the answer is a record, not a reconstruction.

Rollback is a first-class path, not a fire drill. A failed gate does not strand a half-merged change. It returns the increment to its prior known-good state, and the failure itself becomes a Decretum entry: what was attempted, which lens caught it, and why it was refused. A rejected change teaches the system something, and the lesson is durable.

Blast radius is contained by the same isolation that scopes an agent’s work. Because changes move in small, gated increments rather than large speculative merges, the surface a single bad change can touch is bounded before it is ever proposed. And because every action is attributable to a specific actor and decision, an incident review starts from a record of who did what and when, not from an archaeology dig. Observability here is not a dashboard bolted on at the end. It is a property of a system where nothing moves untyped and nothing decides unrecorded.

The eval loop

The routing table the other essay described is not a static opinion. It is the output of a harness that runs continuously, and the harness is where “which model” stops being a debate and becomes a measurement.

A candidate change is routed to more than one model, chosen by the failure mode the task most needs guarded. Their outputs are compared, and the first signal the harness reads is disagreement. When models with different jobs converge, that is weak evidence, because agreement is cheap and a shared blind spot looks exactly like consensus. When they diverge, that is where a human or an arbiter looks, because a disagreement is a claim that something is wrong, raised by an actor with no stake in the original work. Agreement does not buy a change a pass. It still enters the gate.

Behind the disagreement signal is measurement against a baseline. Every model and every routing choice is scored against task suites that stand in for the work we actually do, and against the delivery gates themselves, which are the only benchmark that counts in this domain: did the output survive adversarial review, clear the security scans, and hold accreditation. We watch latency and cost alongside correctness, because a model that is marginally better and materially slower or more expensive is not obviously better at all. And we watch for regression and drift over time, because a model’s behavior is not fixed, and a routing choice that was right last quarter can quietly stop being right.

A model earns a new seat, or loses one, on that evidence. Switching is a versioned change to the routing table, reviewed like any other change, not a leap of faith taken because something new is in the headlines. The table is an artifact with a history, and every entry in it can be traced to the measurement that put it there.

The loop also protects against a subtler failure than a bad model: a good model quietly rotting in place. A routing choice that goes unmeasured becomes a habit, and a habit is how you end up trusting a model on a task it stopped being good at three releases ago. Running the eval continuously, against suites that mirror real work rather than public benchmarks, means the table is never older than the last increment that passed through it. The measurement is not a launch gate you clear once. It is the standing condition for every model keeping its seat.

Humans decide, the bench ships

Read the three diagrams together and the argument is one shape. Coordination that is typed and recorded. Isolation an agent cannot escape. Gates that answer to people. An eval loop that trusts disagreement over confidence. None of it depends on which model is behind the curtain this month. That is the point. The model is the commodity, and the architecture is the thing that carries the guarantees.

This is why I can say the same sentence at the start and the end and mean it as engineering, not slogan. Humans decide, the bench ships. The deciding is real: it lives in the Change Control Board, in the gates agents cannot reach around, in the routing changes reviewed before they take effect. The shipping is real too, and it is fast, because the parts that were never judgment have been handed to the machine and the parts that were always judgment have been kept by people. Hand this to an ISSO and every step has a name. That is what governed means.

  • Dayel Ostraco

    We Don't Have a Favorite Model. We Have a Routing Table.

    The question I get is which model is best. It is the wrong question. In our governed federal factory the first filter is the boundary, the second is failure mode, and the eval that matters is disagreement. The model is a commodity. The routing is the craft.

     · 7 min read

  • Dayel Ostraco

    Compliance Is a Byproduct, Not a Phase.

    Federal software treats compliance as a phase you survive at the end. I built Colophon to invert that: a governed AI agent bench that emits the RMF artifact set as a byproduct of every shipped increment. Humans decide, the bench ships.

     · 8 min read

Back to Insights