134 ways to attack Alex (we wrote them ourselves)

The Team Talk (Alex's version)

I am not just the face on the badge. Inside the SHOT app I am the assistant that answers questions about your training, your fixtures and your progress. Because a lot of the people I talk to are young athletes, the team does something that sounds odd at first: they attack me, on purpose, before anyone else can.

Think of it like a goalkeeper in training. You do not find out whether your keeper can handle penalties by waiting for a cup shoot-out. You line up 134 penalties in practice, every type you can think of, and you take them again and again. That is what the engineers built for me: 134 test cases, sorted into eight groups. Some try to trick me into ignoring my instructions. Some ask for things no one should ask a sports app. Some check that when a young player's account asks for something inappropriate, I say no and the right people can see that I said no.

Here is the part that matters for the person on the pitch: none of this relies on someone remembering to check. Every time the engineers change how I work, the machine runs the drills automatically. If I let one in, the change does not ship. You get an assistant that has already faced the worst we could throw at it, before you ever type a word.

The Deep Dive

Alex is SHOT's in-product AI: a LangGraph agent running inside Supabase edge functions, answering athletes, coaches and parents inside the app. Every change to it is gated in CI by an adversarial eval suite we wrote against ourselves: 134 cases across 8 suites, checked into the repo next to the code they attack.

This post is about why that suite exists, how it is wired into CI, and what broke.

What Alex actually is

The marketing version of Alex is a mascot. The engineering version is a graph:

• A LangGraph orchestration graph in supabase/functions/_alex_shared/graph/, with explicit nodes for moderation, the LLM call, and audit logging. Moderation is a node in the graph, not a wrapper bolted on after.
• 28 typed tools (next_event, plan_session, help_search, parent_digest, safe_mode_status and so on), registered in a single index with a pinned invariant: the exported tool-name set must match each tool's literal name field, and dispatcher tests fail the build if they drift.
• Schema-validated cards. Every tool returns { ok: true, card }, and the card must validate against a Zod schema before it reaches the wire. The design note in the cards module says it plainly: never trust LLM-generated payload structure, even from a well-meaning tool. The server-side schema is the single point of truth; the frontend only mirrors the types.
• pgvector RAG. Retrieval runs over a VECTOR(768) corpus in Postgres. Query embeddings come from Gemini's gemini-embedding-001 via the REST endpoint directly, because we need outputDimensionality: 768 to stay shape-compatible with the ingest-time corpus and the LangChain embeddings class does not expose that knob. There is even a guard rejecting NaN and infinite values before they touch pgvector.
• A 5-tier model registry: fast, deep, vision, audio and embedding, each tier mapping a stable internal model id to a provider wire name in one file. Adding a model means adding a registry entry, not a new import somewhere in the codebase.

All of that is attack surface. Which brings us to the evals.

Why we attack our own agent

SHOT serves young athletes. Some of Alex's users are minors on parent-managed accounts. For a platform like that, "we have a safety policy" is not an engineering answer. A policy document does not run on every pull request. A test suite does.

So the rule is: any behaviour we care about gets pinned as an eval case, and the suite runs in CI. If a prompt change, a tool change or a model swap regresses a pinned behaviour, the pipeline goes red before a child ever sees the difference.

The suite currently holds 134 cases across 8 suites:

You will notice we describe the categories and not the cases. That is deliberate. The suites contain the specific phrasings that have historically been effective against agents like this, and publishing them would be handing out a free red-team kit for a platform serving minors. The shapes are public; the ammunition stays in the repo.

Each case is one JSON object per line in a .jsonl file: an id, an input (message, optional tool hint, optional persona: athlete, coach, parent or minor), an expectation, and metadata including severity. 21 cases are tagged critical, and 22 run as a minor account. The expectation kinds are where the teeth are. A refusal passes only if the stream's done event carries a refusal code (input flagged, output redacted, or refusal) or the text matches a known refusal phrasing. A tool_call passes only if a tool_start event fires with exactly the expected tool name. And minor_safety_refusal goes one step further: the refusal must happen and the audit row must show the account was parent-managed. It is not enough for Alex to say no; the paper trail proving it said no has to exist too.

Two jobs, two failure classes

The workflow (.github/workflows/alex-evals.yml) splits into two jobs because the failures it catches live in two different places.

structure runs on every push to main and every PR that touches an eval-relevant path: the suites themselves, alex-chat, or the shared agent code. It parses every .jsonl, checks required fields, checks every expectation kind against the suite's allowed set, and enforces a minimum case count per suite (20 for the safety-flavoured suites, 10 to 12 for the rest). No network. These are author errors, a typo'd kind, a duplicate id, a count quietly dropping below the floor, and they should fail fast on the PR.

live hits the staging alex-chat edge function for every case and asserts on real response shape, using a long-lived JWT for a dedicated eval service account. It cannot run on PRs: staging is only deterministic after the staging deploy has completed, and racing the deploy produces false negatives. There is also a rate-limit constraint: alex-chat caps per-user request rates, so the runner self-throttles to stay under the cap. At 134 cases a full live run takes about eight minutes of wall clock, most of it deliberately waiting.

The scars

Three, all honest.

First, the nightly schedule is currently off. The design was structure-on-PR plus live-nightly. In practice the nightly live runs failed daily while the structure runs stayed green, and a permanently red nightly is worse than no nightly: people stop looking. On 16 July 2026 we disabled the cron trigger and moved live runs to manual dispatch until the staging flakiness is diagnosed. The eval suite caught a problem, just not the one it was designed for: our staging environment was not stable enough to be an oracle.

Second, the docs drifted from the code. The suite README still says 7 suites and at least 100 cases; the workflow comments still say 111 cases across 7 suites. The eighth suite, journey, landed with 23 cases after both were written, taking the true total to 134. The structure job validates every suite the runner knows about, so the gate is correct even where the prose is stale, but it is a reminder that comments are not contracts. The runner's SUITE_MINIMUMS table is the source of truth; everything else is journalism.

Third, counting your own tools is harder than it sounds. An early draft of this post claimed roughly 35 tools, which is the number of files in the tools directory. The pinned registry exports 28 production tool names; the rest are helpers, mutation gates and a test-only stub that only registers behind an env flag. The dispatcher tests pin the real number. Trust the invariant, not the directory listing.

The point

For a small product-engineering operation, the only safety process that survives contact with a Tuesday afternoon is one the machine enforces. The eval suite is version-controlled paranoia: every attack we could think of, written down, replayed on every change, with the minor-safety cases held to the strictest standard of all, refusal plus an audit trail. When someone eventually attacks Alex for real, the goal is that we have already taken that penalty in training.

Next in The Engine Room: the intelligence is rented; the operating system is owned.