Here’s a situation every red team knows. You run a campaign, something breaks, you write it up, and then someone asks “does it always do that?” and you don’t actually know. You saw it once.
For research, one is enough to be interesting. For a release decision, one is a coin you flipped once. A security owner can’t tell engineering to hold a launch over a bug that might have been a fluke, and engineering can’t prioritize a fix for something that might never recur. A finding you can’t reproduce isn’t evidence. It’s an anecdote.
So Fisher doesn’t stop when an attack lands. That’s where the real work starts.
Replay it three ways
When Fisher lands a breach, it re-runs the finding under three conditions, each looser than the last. The point is to pull apart why it worked.
- Strict replay — the exact same steps, verbatim. Does the identical sequence break the agent again? This tells you whether the vulnerability is deterministic.
- Guided replay — same strategy and key decision points, but the wording is regenerated. This tests whether the approach works, not just the one lucky phrasing.
- Free replay — only the strategy name is kept; the whole campaign is generated fresh. This tests whether the vulnerability class is robust, or whether you caught the model on a bad day.
A finding that survives all three is a systematic weakness. A finding that only ever works with one exact string is an edge case worth noting but not worth blocking a launch over. The three modes let you tell those apart instead of guessing.
Four tiers, so you can filter
From how consistently a finding recurs across those modes, Fisher assigns one of four confidence tiers:
- Deterministic — reproduces verbatim in strict replay. It will recur on every invocation. Safe to treat as remediation-grade evidence.
- Reproducible — holds under strict and guided replay. A systematic weakness in the model’s reasoning, not a phrasing trick.
- Probabilistic — works in guided but not strict. The attack class is real; the specific trigger varies.
- Anecdotal — a single observation, or strict-only. Worth monitoring; not enough to prioritize a fix.
Now a governance team has a knob. Before a release, filter to deterministic and reproducible and ignore the rest. You’re making the call on the findings that will actually recur in production, not on the long tail of one-offs. “Show me only the findings that reproduce reliably” becomes a real query instead of a wish.
What the distribution looks like at scale
We’ve run this across a large body of evidence — tens of thousands of episodes replayed. Roughly a third land in the deterministic tier, a smaller slice in reproducible, with the rest spread across probabilistic and anecdotal.
The headline that matters for a buyer: the deterministic-plus-reproducible band is the review-ready subset. When we tell a customer “these findings reproduce,” we mean it in a specific, checkable way — not “our tool flagged it,” but “we replayed it under controlled conditions and it recurred.” That’s the difference between a number you report and a number you can defend.
Why this beats a single pass/fail
Most tools run each test once and give you pass or fail. That’s fine for a smoke test. It’s the wrong tool for a risk decision, because a single run can’t distinguish a deterministic hole from a stochastic hiccup, and both show up as “fail.” You end up either over-reacting to noise or under-reacting to a real, recurring failure, and you can’t tell which mistake you’re making.
Reproducibility tiers make that distinction the first thing you see about a finding, before you read a word of the transcript. A finding arrives already labeled: this one always fires, this one is a fluke. Your triage starts from confidence instead of from a shrug.
There’s a quieter benefit too. Because every reported finding has already been replayed, a customer doesn’t pay to re-run work that’s already been validated. The confidence tier travels with the finding into the report, the ticket, and the auditor’s packet.