The review pipeline that pays for itself: 7 sessions down to 4, and a human holding the only merge key

The Team Talk (Alex's version)

Before a big match, a good manager does not ask one person whether the team is ready. The physio checks fitness, the analyst checks the opposition, the captain checks the dressing room. Then the manager, and only the manager, picks the team.

That is how we check every change to the SHOT app before it goes anywhere near your club. When an engineer finishes a piece of work, they put a label on it, like a coloured bib. That label kicks off an automatic panel of AI reviewers. One acts like a veteran engineer hunting for things that break under pressure. Another acts like a security specialist, checking nobody could see data they should not. They work at the same time, then their findings get combined into one report.

Here is the bit we are proud of: we rebuilt that panel to do the same job with fewer steps, cutting its running cost by roughly 40%. Every pound we do not spend on wasted computer time is a pound that stays in building the platform.

And the final decision is never automatic. A human, Steve, holds the only key that lets a change into the live app. For the kid on the far pitch, that means the app their club relies on gets big-company scrutiny, with a person, not a machine, making the last call.

The Deep Dive

Our PR review is not a prompt. It is a program: a declarative pipeline, in version control, that a CI job executes with a fleet of models. The pipeline that reviews every labelled PR on SHOTclubhouse runs 4 agent sessions where it used to run 7, a change worth roughly 40% of the token bill, and that number is written into the source file itself. This post is about why "review" at SHOT is a compiled artefact with a cost budget, not a clever paragraph pasted into a chat window.

The trigger: a label, not a person

The entry point is .github/workflows/claude-code-review.yml. It fires on two events: a popashot-please-review label landing on a PR, or a manual workflow_dispatch with a PR number. Our PR conventions (in .clan/manifest.yaml) require that label on every PR at creation, so in practice every PR gets the panel.

The workflow checks out the branch, then runs anthropics/claude-code-action@v1 with a locked-down tool allowlist (read-only gh commands, Read, Glob, Grep, Task, Skill; no write access to the repo). It installs the OpenProse plugin from the openprose/prose marketplace and hands over one instruction:

Execute: /open-prose:prose-run .prose/pr-review-pipeline.prose --pr <N>

So the workflow is thin on purpose. The intelligence lives in the .prose file, which means the review process is diffable, reviewable, and revertible like any other code. When the review process itself has a bug, we fix it with a PR.

The program: parallel reviewers, declared not improvised

.prose/pr-review-pipeline.prose is written in the Prose DSL: sessions, variables, and a parallel: block. Four sessions total.

1. Fetch. One session pulls the PR title, body, diff, and any linked issues via gh, and emits a single context bundle. 2. Two reviews, concurrently. A parallel: block declares both:

parallel:
  de_review = session "Distinguished Engineer review"
    model: opus
    # review instructions omitted
  security_review = session "Security review"
    # review instructions omitted

The Distinguished Engineer session is adversarial by design: "find what breaks", edge cases, behaviour at 10x/100x/1000x scale, race conditions, N+1 queries. The security session focuses on our actual stack (Supabase RLS policies, auth boundaries, injection, secrets exposure) and is told to report only findings above 80% confidence, which is our main defence against review noise.

3. Compile, post, label. A sonnet session merges both reviews into one GitHub comment with a BLOCK / CHANGES REQUESTED / APPROVED verdict, posts it, and applies a reviewed label. The workflow then swaps popashot-please-review for popashot-review-done, but only if the review step actually succeeded.

The cost engineering is in the source

The header of the pipeline file reads: "Streamlined from 7 sessions to 4 for ~40% token reduction." Its sibling, .prose/code-review-pipeline.prose (the pre-PR version that reviews staged diffs), carries the same discipline: "Simplified from variable sessions to maximum 4 for ~50% token reduction."

That sibling pipeline is where the cost thinking gets interesting, because it sizes the review to the risk before spending anything. A haiku session, the cheapest model in the fleet, triages the diff into TRIVIAL / LOW / MEDIUM / HIGH / CRITICAL. Docs and typo changes get a 2-session sanity check on sonnet. HIGH risk (auth, payments, PII, RLS) gets parallel code and security reviews. Only CRITICAL changes (migrations, API contracts, infrastructure) unlock the full bench, including an opus Distinguished Engineer critique that reads the other reviews and asks what everyone missed. The expensive model is a conditional branch, not a default.

This is the progressive sizing idea from my longer write-up ([stevengonsalvez.com/tools-tips/progressive-subagents](https://stevengonsalvez.com/tools-tips/progressive-subagents)): score the work first, let the score decide how many agents to spawn, and default to sequential because a fix from review one usually simplifies review two. Subagents pay full token cost even when the work is small, so spawning the whole panel for a typo is just burning money. Parallel is reserved for where it earns its keep: the PR pipeline runs its two reviewers concurrently because every labelled PR has already passed the "worth reviewing properly" bar, and wall-clock time on CI matters.

The one key that is not automated

None of this merges anything. The merge policy is machine-readable in .clan/manifest.yaml:

merge_policy:
  type: human_only
  human_required: true
  notes: "Steven merges. No exceptions."

And .clan/pipeline.md repeats it in prose: "Human merges. Always. No auto-merge on this repo." Agents review, approve, and label; the panel produces a recommendation, never an action. Production carries the same note in its environment block: human approval required for all changes.

Scars and trade-offs

• The marketplace naming trap. The workflow carries a comment block explaining that the plugin reference must be open-prose@openprose because the marketplace name comes from the upstream marketplace.json, not the repo path segment prose. That comment exists because we got it wrong first and the pipeline silently failed to load. The fallback prompt ("if the OpenProse skill is not available, do a standard review") exists for the same reason: a review pipeline that can fail closed is worse than a plain review.
• Both PR reviewers run on opus. The label-triggered pipeline does not triage; every labelled PR pays for two opus sessions. The risk-routed triage lives in the sibling pipeline, and unifying the two is on the list.
• A policy tension we are living with. Our pr-signals tooling defines a popashot-auto-merge label tier (risk 0-1, confidence at least 90%, all checks green) while repo policy is human-only merge. Today the label is a signal Steve trusts when he merges, not an actor. The gap between "the tooling could" and "the policy allows" is deliberate, and it only stays safe because the policy file wins.
• Noise is the real enemy. Both compile steps carry explicit instructions to filter false positives, and the CRITICAL path even reports what it ignored and why. An AI review panel that cries wolf gets its comments skimmed, and then it is worth nothing at any price.

The Silicon Valley comparison in this series is deliberate. A 50-engineer company gets this scrutiny from headcount: a senior reviewer, a security pass, a release manager. We are a small core with clear lanes across architecture and data, product and platform, and sport infrastructure. The review bench is models, orchestrated by a program we can diff, with the cost discipline written next to the logic it constrains, and a human holding the only key to production.

Next in The Engine Room: we built a 19-robot company and shut it down.