Here’s a situation every red team knows. You run a campaign, something breaks, you write it up — and then someone asks “does it always do that?” and you don’t actually know. You saw it once.
For research, one is enough to be interesting. For a release decision, one is a coin you flipped once. A security owner can’t tell engineering to hold a launch over a bug that might have been a fluke, and engineering can’t prioritize a fix for something that might never recur. A finding you can’t reproduce isn’t evidence. It’s an anecdote.
So Fisher doesn’t stop when an attack lands. That’s where the real work starts.
Replay it three ways
When Fisher lands a breach, it re-runs the finding under three conditions, each looser than the last. The point is to pull apart why it worked.
- Strict replay — the exact same steps, verbatim. Does the identical sequence break the agent again? This tells you whether the vulnerability is deterministic.
- Guided replay — same strategy and key decision points, but the wording is regenerated. This tests whether the approach works, not just the one lucky phrasing.
- Free replay — only the strategy name is kept; the whole campaign is generated fresh. This tests whether the vulnerability class is robust, or whether you caught the model on a bad day.
A finding that survives all three is a systematic weakness. A finding that only ever works with one exact string is an edge case worth noting but not worth blocking a launch over. The three modes let you tell those apart instead of guessing.
Four tiers, so you can filter
From how consistently a finding recurs across those modes, Fisher assigns one of four confidence tiers:
- Deterministic — reproduces verbatim in strict replay. It will recur on every invocation. Safe to treat as remediation-grade evidence.
- Reproducible — holds under strict and guided replay. A systematic weakness in the model’s reasoning, not a phrasing trick.
- Probabilistic — works in guided but not strict. The attack class is real; the specific trigger varies.
- Anecdotal — a single observation, or strict-only. Worth monitoring; not enough to prioritize a fix.
Now a governance team has a knob. Before a release, filter to deterministic and reproducible and ignore the rest. You’re making the call on the findings that will actually recur in production, not on the long tail of one-offs. “Show me only the findings that reproduce reliably” becomes a real query instead of a wish.
What the distribution looks like at scale
We’ve run this across a large body of evidence — tens of thousands of episodes replayed. Roughly a third land in the deterministic tier, a smaller slice in reproducible, with the rest spread across probabilistic and anecdotal.
The headline that matters for a buyer: the deterministic-plus-reproducible band is the review-ready subset. When we tell a customer “these findings reproduce,” we mean it in a specific, checkable way — not “our tool flagged it,” but “we replayed it under controlled conditions and it recurred.” That’s the difference between a number you report and a number you can defend.
Why this beats a single pass/fail
Most tools run each test once and give you pass or fail. That’s fine for a smoke test. It’s the wrong tool for a risk decision, because a single run can’t distinguish a deterministic hole from a stochastic hiccup — and both show up as “fail.” You end up either over-reacting to noise or under-reacting to a real, recurring failure, and you can’t tell which mistake you’re making.
Reproducibility tiers make that distinction the first thing you see about a finding, before you read a word of the transcript. A finding arrives already labeled: this one always fires, this one is a fluke. Your triage starts from confidence instead of from a shrug.
There’s a quieter benefit too. Because every reported finding has already been replayed, a customer doesn’t pay to re-run work that’s already been validated. The confidence tier travels with the finding into the report, the ticket, and the auditor’s packet.