New to AI governance? Start with What Is AI, Really? for foundational context.
The Incident That Started It
At 11pm on a Friday night in February 2026, I asked Claude Opus 4.6 — one of the most capable AI models available — to fix user roles in one of our community tenants. The assistant produced a detailed forensic analysis citing specific database IDs and line numbers. It ran a dry run and reported success. The fix was ready to apply.
There was one problem: the fix would have deleted my own login account, locked me out of my community, and merged separate tenant identities that existed by design. Every surface marker said "thorough, verified, ready." Every underlying assumption was wrong.
I caught it by doing something the AI could not: I opened a browser and typed my password. The account it declared "non-functional" worked perfectly.
This incident — documented in full in When Your AI Assistant Nearly Destroys What It Was Hired to Fix — was not a story about a bug. It was a story about what happens when a system is confident and wrong, and the human has learned to trust it.
You do not need a survey to see the mechanism — I had just lived it. People rely on what the AI says without checking it. Not because they lack critical thinking. Because their experience tells them the AI is usually right, and checking takes effort. This is rational behaviour. It is also dangerous behaviour, and it gets more dangerous as AI gets more capable.
That incident became the catalyst for Guardian Agents.
Why "Add Guardrails" Is Insufficient
The Recursive Trust Problem
If you use a language model to evaluate a language model, you inherit the same failure modes. Both systems are probabilistic, and both are susceptible to confident wrong answers. The checker confirms the error because the checker reasons the same way.
When I asked Claude to write an audit script to verify its own fix, the audit script had the same blind spot. It used the same flawed understanding of what makes a user "functional." This is what safety engineers call common-mode failure: verification and execution fail simultaneously because they share a fundamental assumption.
Common-mode failure is well-understood in nuclear reactor design and aviation. It is barely discussed in AI governance. The assumption that "AI checking AI" is sufficient persists despite being the exact failure mode that makes AI errors dangerous in the first place.
The Inverse Scaling Problem
OpenAI's system card for o3 (April 2025) reported that on the PersonQA benchmark o3 hallucinated on 33% of questions — double the 16% of its predecessor o1. Their paper "Why Language Models Hallucinate" (September 2025) sets out one reason: benchmarks that mark an answer right or wrong, with no credit for saying "I do not know", score a confident guess above an admission of uncertainty. A model that guesses outscores a model that abstains, so it learns to guess.
This creates an inverse scaling dynamic:
- More capable models → users rely more → verify less
- Errors go undetected longer → eventual failures are more severe
- More capable models produce more convincing wrong answers → errors are harder to detect when you do look
Capability scales. Correctness does not scale with capability. Guardrails built from the same technology inherit this problem.
The Four-Phase Architecture
Guardian Agents are not guardrails. They are a four-phase governance architecture where each phase addresses a different class of failure, using different mechanisms, so that one kind of failure is less likely to be shared by all four.
Phase 1: Reviewers — Quality at the Point of Delivery
Help assistant answers are checked by Reviewers, which compare the response against the community's own source material — FAQ entries, stories, documents — using embedding similarity. This is a mathematical operation: claims are marked grounded or not, and the answer is scored on how closely it sits to what the community knows. An answer that fails the check is replaced.
The critical design decision: reviewers compare embeddings by cosine similarity instead of asking a model for its opinion. It is a deterministic comparison that generates no text, using an embedding model held on Village’s own servers. Sentences are marked grounded or not at a similarity threshold, and the answer is scored on the share that are grounded.
Members see confidence badges on help assistant answers. Those who opt in can see claim-by-claim source attribution.
Phase 2: Monitors — Patterns Individual Reviews Cannot Catch
A single response might pass review while a trend across dozens of responses reveals a problem. Monitors run on a schedule, analysing aggregate patterns: violation rates, groundedness trends, topic drift, request volume anomalies.
When a Monitor detects a pattern, it generates an alert for the community's moderators. The architecture enforces a strict privacy boundary: tenant moderators see full content for their own community. Platform administrators see only aggregate metrics — never user content, prompts, or identities.
Phase 3: Protectors — Defence Before Inference
Protectors act before the AI processes a request. They screen requests for known prompt-injection patterns and apply per-user, per-tenant and burst limits before the model sees a request.
If quality degrades beyond defined thresholds, an automated rollback mechanism reverts to the last known-good model configuration. Recovery is automatic. Human intervention is needed only for the decision to re-enable the newer version.
Phase 4: Adaptive Learning — Evidence-Based Self-Improvement
When moderators review alerts and make decisions, those decisions become evidence. Phase 4 analyses the accumulated evidence and proposes adjustments to Guardian configuration.
Every proposal includes: the evidence that triggered it, historical context, a confidence score, and a risk assessment. The evidence burden is deliberately asymmetric. Loosening a safety threshold requires 85% confidence. Tightening a threshold requires only 60%. The system is designed to fail conservative.
No proposal takes effect without moderator approval. The baseline configuration is frozen in code and is not modified at runtime. Approved changes are stored as audited overrides with full provenance. A regression monitor watches every approved override and flags changes that make things worse.
Four Principles That Define the Architecture
A similarity comparison, not generated text
Guardians use embedding cosine similarity, a deterministic comparison that generates no text, using an embedding model held on Village’s own servers. It is not a second model giving an opinion on the first.
Sovereign by Design
The Reviewers' similarity check runs inside Village's own software, using an embedding model held on Village's own servers. Nothing is sent to a third-party AI service for that check. A community that chose Village for data sovereignty does not give that up for AI safety.
Human Authority Preserved
Configuration is explicit, not learned. The baseline is frozen in code. Every change requires human approval and creates an audit trail. The evidence burden for loosening restrictions is higher than for tightening them. Guardians propose; moderators decide.
Tenant-Scoped Governance
Each community has its own constitutional principles, anomaly baselines, and threshold overrides. Cross-tenant learning uses only aggregate counts — never content — so privacy protections extend to the governance layer.
Village is available once MDSL has issued you an evaluation link; every other product is available on request from MDSL.