Every morning, robots play a full season

The Team Talk (Alex's version)

Every club has a groundsman who walks the pitch before anyone plays on it. We went further. We built a full copy of SHOT, a practice ground, and filled it with robot players and robot coaches who actually live in it, day after day.

Each morning while you sleep, robot athletes fill in their pre-session check-ins, robot coaches score every player and leave comments, and robot players write their reflections afterwards. They log in with their own accounts and come through the same doors real families use. Nothing is faked behind the scenes.

Then we run 8 health checks on their world. Is the season still rolling forward? Did every finished match get a score? Did every player who trained get their coach's feedback? If a single check fails, a red light goes on before breakfast, and we fix it before any real coach or parent can hit the same problem.

Finally, a robot logs in as the head coach, walks the whole app and takes photographs of every screen, so a human can glance through the morning's pictures with a cuppa.

What it means for the kid on the far pitch: by the time you open the app, something has already played a full day of sport inside it and proved it still works.

The Deep Dive

Most staging environments are graveyards. A snapshot of production data, restored months ago, drifting away from anything real; visited by an engineer the day before a release and by nobody in between. The standard Silicon Valley answer to "how do you know it works?" is a pyramid: unit tests, integration tests, an end-to-end suite against synthetic fixtures, and a staging environment that exists mostly so the deploy pipeline has somewhere to point. All of it tests that the code passes tests. None of it tests that the product still works for a person living inside it, because nobody lives inside it.

We inverted that. Our staging environment is a digital twin of a running sports organisation, and it has residents. Every morning, synthetic athletes and coaches complete the platform's core loop, the evaluation cycle, through the real product API, under their own logins. Then a monitor asserts the twin is coherent, and a browser walks the app as the head coach and photographs what it sees. A red run means the product broke, not that a test got flaky.

The twin itself

The seed (supabase/admin-scripts/allstars-seed/) builds a five-club, five-sport organisation on staging: football, boxing, cricket, padel and rugby. Thirteen named teams plus lightweight extras so every valid evaluation-framework level per sport is exercised, roughly 250 to 300 athletes on full-spec rosters, 12 months of past events and matches, evaluations in monthly blocks of weekly cycles (about 48 per team-year), attendance averaging around 85% with present, absent and late variation, sport-appropriate scorelines on every completed match, and all five of our age-and-consent states represented, from under-13 parent-attached through 16-plus self-managed.

Crucially, the twin does not stand still. A pg_cron job fires at 03:00 UTC daily and rolls the world forward: past events complete, the schedule keeps an always-eight-week forward horizon, the season never runs out. Every row the seed creates carries explicit test markers, teardown deletes strictly by marker, and every script calls a staging assertion before writing anything; the scripts refuse to run if pointed at production. These are admin scripts, not migrations, so they can never auto-deploy.

The morning shift: three jobs, chained

At 06:30 UTC, after the roll-forward, a GitHub Actions workflow (.github/workflows/allstars-twin.yml) runs three chained jobs.

Job 1, simulate. A shell script logs each persona actor in through the auth endpoint with their own credentials, takes the JWT it gets back, and writes through PostgREST exactly as the product's front end would. Athletes on teams with an event today submit pre-evaluation scores; the coach actor scores every athlete on events that recently completed, leaving a comment roughly one time in four; athletes then submit their post-event reflections. The script's header states the rule that makes this honest:

ZERO service_role writes in the actor path — every eval write above goes
through PostgREST with the acting persona's JWT (real RLS).

No superuser back door. If row-level security is wrong, if the auth flow breaks, if the evaluation service's write path regresses, the actors fail the same way a real teenager on a real phone would fail. The privileged Management API token is used only for read-only orchestration (which events exist, which rows need scoring) and the row-provisioning plumbing the product's own publish trigger would normally do. Every run records its counts into a sim_runs table, which the next job reads.

Job 2, invariants. This is the actual gate, and it runs with if: always(), so a crashed simulator does not skip the monitor. The workflow comment is blunt: the monitor runs even if the simulator failed, because that is itself a signal. One SQL round trip evaluates eight invariants:

• I1: the maintenance cron left a heartbeat in the last 26 hours.
• I2: no team's published schedule has fallen below the six-week forward horizon.
• I3: no event that finished more than 36 hours ago is still unprocessed.
• I4: no evaluation marked as missed is carrying scores (coherence, not just liveness).
• I5: no completed match is missing a scoreline.
• I6: the simulator recorded a clean run in the last 26 hours.
• I7: no actor-team event that completed more than 30 hours ago still has unscored evaluations.
• I8: the club and team counts match the seeded shape.

Any breach exits non-zero and the run goes red. RED means the twin broke: the season stopped rolling, the actors could not complete the loop, or the data went incoherent. There is no "rerun it, probably flaky" category here, and that took deliberate work (see the scars below).

Job 3, coach-eye. A standalone Playwright config logs into the staging app as the head coach and walks the twin the way a human would: all five clubs visible, a club dashboard opens, teams render, a roster shows named players. Assertions are text-anchored on seeded club and team names rather than CSS selectors, so they survive UI refactors. Every step screenshots, full page, and the images upload as workflow artifacts with 14-day retention. A human can flick through the morning's photographs in under a minute.

Why the gate is data, not screens

Notice the ordering: the browser walkthrough is the last job and the least trusted one. That is deliberate, a lesson paid for elsewhere in our tooling. Dynamic click-through validation, where an agent or a generated script drives the browser and declares success, measured under about 70% reliable in our experience, and its assertions tend to fire close to DOM-load. A page that renders its shell and none of its data can pass a screenshot test. An empty roster looks a lot like a loading roster.

So the twin's authoritative signal is the invariants job, eight assertions on the database that the product's behaviour must have produced, and the coach-eye exists for the failure modes SQL cannot see: a broken login form, a dashboard that white-screens, a layout that eats the team list. Data proves the loop ran; the screenshots prove a human could have watched it run. Neither alone is "it works".

Where the twin sits in the wider rig

The twin is one layer of a testing architecture that has grown its own geography. The main Playwright config currently defines 18 projects. The core pattern is lanes by data lifecycle: Lane 1 creates fresh users and data every run (real signup simulation, slowest), Lane 2 runs against pre-seeded data under a stored coach login (fast event and evaluation flows), Lane 3 tests deadline logic against backdated historical data (fastest). Around those sit the Red Route certification suites: an ephemeral-registration lane where every account is created, tagged and torn down per run, and a five-persona operational lane, coach creates and publishes, athletes and parents verify visibility, phases strictly serial.

All of that is still testing in the conventional sense: a run starts, asserts, and ends. The twin is a population. The actors' data accumulates, the season rolls, and the question asserted is not "did this flow pass" but "is this world still alive and coherent after another day of use".

Scars

Two honest ones. First, on 12 July the Management API had a flaky minute and a transient non-array response masqueraded as a failure. A monitoring system that cries wolf trains you to ignore red, which destroys the entire premise of "red means broke". Both scripts now retry three times with backoff before concluding anything, and the code comments carry the date of the lesson so nobody optimises the retries away.

Second, smaller and cheaper: the simulator's header comment says it scores events completed "in the last 48h" while the SQL underneath scans a seven-day window with the 30-hour invariant doing the real policing. The comment lost a race with the code. We are publishing that rather than quietly fixing it first, because the gap between what a system says it does and what it does is exactly what this whole workflow exists to catch.

Most companies test that the code passes tests. We also make synthetic people live in the product every day, and we only believe the product works because they can.

Next in The Engine Room: the review pipeline that pays for itself.