Two robots, one broken signup: what 227 rage clicks taught us about frontier models

The Team Talk (Alex's version)

Imagine your club has two scouts. One watches the match in front of them and spots every player struggling right now. The other goes back through months of footage and works out why the problems started. Neither scout is better. You need both, and you need to know which one to send to which game.

That is how we run the AI systems behind SHOT. When something breaks, we do not guess which AI to trust. We have tested them, head to head, on real problems from our own platform.

One night a player could not sign up on their phone. They tried for 8 minutes and tapped the screen 227 times before giving up. One AI found the cause on its own: a single full stop typed in the wrong place in an email address. The other AI needed a human to step in three times, and it declared victory too early.

We learned which scout to send first, and we kept both on the books. What it means for the kid on the far pitch: when signup breaks at 10pm, it gets fixed before morning training, because the right robot was already on the case.

The Deep Dive

We do not benchmark frontier models on puzzle sets. We benchmark them on our own production incidents, because SHOT is a live consumer platform and the only test that matters is whether the model can fix the thing that broke for a real family last night. Over two incidents in spring 2026 we ran Anthropic's Opus and OpenAI's GPT head to head on identical incident data, identical tooling, identical knowledge indexes. The full technical write-ups are on Steve's personal blog ([part one](https://stevengonsalvez.com/blog/opus-vs-gpt-on-real-ops), [part two](https://stevengonsalvez.com/blog/opus-vs-gpt-on-real-ops-part-2)); this is the SHOT view of what those experiments changed about how we operate.

Incident one: the orphaned accounts

Two child accounts sat in production for a week without the database row linking them to a parent. On a platform where parental consent is the trust model, that is not a cosmetic bug. A parent flagged it through support: "unable to see my child account."

We gave the identical incident to three agent cells: GPT-5.5 in two different harnesses and Opus 4.7 in a third. Same knowledge index, same skills, same MCP servers. The privacy boundary held throughout: only UUIDs crossed into the agents, and any column tagged PII in the schema metadata was stripped from every projection, so the models never saw personal data even when they asked for SELECT *. That guardrail is not incidental on a platform serving minors; it is the precondition for running these experiments at all.

The two model families did not just perform differently. They thought differently.

• Opus went predictive. It assessed present risk, surfaced additional orphaned accounts beyond the two reported, and proposed a concrete fix: run the existing account-linking function with the specific parent IDs it had identified. A band-aid, applied in minutes.
• GPT went causal. It walked the migration files and reconstructed a timeline: a 6 March schema cutover had made the relationship table canonical but deferred the backfill, and a later account deletion and re-registration under a new UUID left the children permanently orphaned. A root cause, with the migration gap to close.

The control mattered: both GPT cells, in different harnesses, pursued causal analysis. Only Opus went predictive. The reasoning style belonged to the model, not the scaffolding around it. And here is the operational punchline: we shipped both answers. Opus's fix restored the affected families within minutes; GPT's diagnosis closed the gap so it could not happen again. Neither alone was the resolution.

Incident two: the signup that would not complete

A month later, a SEV-3: a user could not finish signup on Android. The session telemetry showed 8 minutes and 227 clicks, including rage clicks. The nasty part: a failed signup never fires the identify event, so there was no email, no username, no user ID in analytics. The investigation had to start from almost nothing.

This time we ran Opus 4.8 against GPT-5.5 on the live investigation, and measured the thing that actually costs money at 2am: human intervention.

• Opus: 0 nudges. It worked out on its own that an abandoned signup leaves no identity trail, triangulated the session from timestamps and events, decoded the session-replay blobs, and traced the missing reset email across database triggers and the mail provider to prove it was never sent. Root cause found autonomously: a Gmail dot-variant typo. The user had typed their address with the dot in a different position; Gmail treats the two as the same inbox, our auth database treats them as different accounts, so the "existing" account could never receive its email.
• GPT: 3 interventions. It needed steering towards the replay tooling, and it stopped at an HTTP 200 response from the email endpoint and called it delivered. That 200 is anti-enumeration politeness, not proof of delivery; the API answers success on purpose so attackers cannot probe which emails exist. Trusting it is exactly the kind of plausible-but-wrong conclusion that burns an on-call engineer.

As the source post puts it: the scarce resource at 2am is human attention, not tokens.

What this changed at SHOT

We are model-agnostic on principle and specialist by evidence. Both families stay in the toolbox, and we route by task shape:

• Blast-radius and remediation work (who else is affected, what do we run right now) routes to the model that demonstrated predictive, present-tense reasoning.
• Archaeology (how did this state come to exist, which migration or cutover caused it) routes to the model that demonstrated causal, historical reasoning.
• Unsupervised overnight investigation routes to whichever model most recently proved it can run without nudges, because interventions are the real cost.

This is what "Silicon Valley practice from a small UK product-engineering team" means in the concrete. We cannot staff a 24-hour ops rota, so the operating system around the models has to be the rota: PII-stripped data projections, shared knowledge indexes, multiple harnesses, and measured head-to-heads on our own incidents instead of vendor benchmark charts. The models are rented; the routing evidence is ours.

The scars

Honest costs from these two experiments:

• The orphaned accounts sat for a week before a parent's support message surfaced them. The agents found the cause fast; our detection did not. Consent-link integrity is now monitored rather than assumed.
• The Gmail dot-variant collision was our bug before it was anyone's investigation. Two addresses that are one inbox in Gmail were two accounts in our auth database. The model found it; we still had to own it.
• Running three parallel agent cells on one incident is not free. It costs tokens, setup, and discipline in keeping the inputs identical. We think the routing evidence pays for it; on a quieter platform it might not.
• A 200 status code fooled a frontier model. It will fool a tired human too. Verification steps that prove delivery, not acceptance, are now part of the incident playbook.

Next in The Engine Room: every morning, robots play a full season.