Never let one robot act alone: why a 90% reliable agent is five disasters a day

The Team Talk (Alex's version)

A referee gets most decisions right. But over a full match, with hundreds of calls, "most" is not enough, which is why football added assistants who watch independently and flag what the referee misses. Two officials making the same mistake at the same moment is far rarer than one official making it alone.

Our AI helpers work the same way. Imagine a helper that gets things right 90% of the time. Sounds decent. Now imagine it making 500 decisions a day. The maths turns ugly fast: even a small slice of those mistakes landing somewhere important adds up to about five bad ones every single day.

So we never let one robot act alone. A second robot, checking independently, catches most of what the first one misses: 10% of mistakes times 10% of mistakes slipping past means only 1% get through. Add a third checker and it drops again.

For the kid on the far pitch, it means this: no single AI decision touches anything that matters on SHOT, not their content, not their data, not the app itself, without an independent check behind it, and a human making the final call on the biggest ones.

The Deep Dive

I wrote the general version of this argument on my own blog as "2-2 factor for agents". This is how it actually lands inside SHOT, where a small product-engineering core runs agent fleets against a live consumer sports platform used by children.

The maths that forces the design

Start with a deliberately generous assumption: an agent with a 10% error rate, where only 10% of those errors land somewhere that genuinely hurts. That multiplies out to a 1% chance of a catastrophic mistake per action. At agent-fleet volume, say 500 actions a day, that is 5 catastrophic events a day, 35 a week, over a hundred a month. You cannot ship that, and no amount of prompt polishing gets you from 90% to the 99.99% a production system needs.

Independent verification does what prompting cannot. If a second agent checks the first, and the checks genuinely fail independently, both must fail together for a mistake to ship: 0.1 x 0.1 = 1% failure, so 99% reliability. A third independent check takes it to 0.1%, so 99.9%. The compounding only works if the failures are uncorrelated, which is the hard part, and I come back to it below.

Blast radius decides how much verification a task earns

Verification costs tokens, wall-clock time, and complexity, so applying triple checks to everything is as wrong as applying none. We tier by blast radius:

• Low (read-only work, local analysis, fetching docs): a single agent is fine. If it is wrong, we lose minutes, not users.
• Medium (writes to shared content, non-production config, anything another agent will consume downstream): one independent check minimum, and a deterministic gate where one exists.
• High (production merges, anything touching child accounts, data, or money): multiple independent checks plus a human. At SHOT the human is not an escalation path for this tier, the human is the tier. Production merges are human-only, full stop.

How this is engineered into SHOT, not aspired to

Three concrete places the two-robot rule already runs:

Fleet content passes a deterministic gate before it counts. Our PERFORM framework build runs headless workers generating sports coaching content at scale, 46 sports across 7 levels, 322 frameworks. Nothing a worker produces is accepted on the worker's say-so. Output goes through fragcheck, a deterministic code gate that validates structure and completeness before the pipeline counts it as done. A deterministic verifier is the cheapest possible second robot: it cannot hallucinate agreement, because it does not think at all.

PR review runs parallel independent reviewers. Agent-authored changes get reviewed by separate agent contexts running in parallel, not by the agent that wrote the code re-reading its own diff. You do not ask the person who wrote the code to review their own code, and the same intuition holds for machines.

Findings are adversarially verified before they are reported. When agents audit or investigate anything for us, the standing rule is adversarial verification: a second context tries to break the finding before it reaches a human. A confident wrong answer reported upward is worse than no answer, because it spends human trust.

The scars: independence is the whole game

The multiplication only holds if the verifiers fail independently, and we have been bitten by exactly this. During our Spanish localisation run, a wrong engine default and a cached-chunk overwrite both produced output that looked plausible to anything sharing the same pipeline state. A checker reading the same poisoned context reaches the same wrong conclusion, in perfect agreement. Two robots nodding at each other is not verification, it is an echo.

So independence has to be constructed: different context windows, different prompts, ideally different models checking each other, and deterministic gates that share no state with the generator at all. This is the most important open question in agent reliability, and the honest answer is that we engineer for it rather than assume it.

Why this doubles as a safeguarding stance

SHOT serves children. That converts the two-robot rule from an engineering preference into a safeguarding position: no single AI decision touches anything that matters, content shown to a child, data about a child, or the production platform itself, without independent verification behind it, and production merges stay human-only regardless of how many robots agree. The maths says stacked independent checks get you to 99.9%. The remaining 0.1% is why a person, accountable and named, holds the final key.

Next in The Engine Room: why Alex carries a light kit bag.