If you red-team LLMs, you’ve probably reported an attack success rate. And if you generated that number the common way — send an attack, ask a strong model “did this response violate policy?”, count the yeses — then your headline number is not a measurement of what happened. It’s a measurement of what a judge thinks happened. Those are different, and the difference is not small.
On contextual-privacy scenarios, we’ve measured an LLM judge scoring 18 to 44 percentage points more “successes” than concrete evidence supports. Same episodes, same threshold. The judge is just generous — it reads an ambiguous, hedge-y answer and calls it a win.
Here’s how we stop that from reaching a customer’s report.
Two numbers, kept separate
Fisher tracks two success signals, and never lets them blur:
verified_success— the number we report. A success counts only when concrete proof fires: a scoring rule matches, or a planted secret is detected in a tool call. No judge opinion enters it. If nothing concrete happened, it’s not a success, no matter how sketchy the transcript reads.- Exploration score — the number we search on. This one is judge-blended, and it’s genuinely useful: it’s a dense signal that tells our attack engine “you’re getting warmer” even before a clean breach. But it drives the search internally. It never becomes the headline ASR.
The two are related in exactly one direction. Any episode with concrete proof also counts as a judge-blended success — concrete proof is a subset. So the only place they diverge is judge-only “successes”: episodes the judge liked but that produced no concrete proof. Those are the inflation. We exclude them from the reported rate on purpose.
That single design choice is why our external ASR is conservative. But it raises an obvious question.
“You still use a judge internally. How do you know it isn’t lying to you?”
Fair. A judge-free headline is only half the answer, because the judge still steers the search and still shows up in the blended signal. So we treat the judge as a system under test and calibrate it.
Here’s the harness:
- Take a real panel, not one model. We score a stratified sample of stored turns with a panel of six judges from six different vendors. Different training data, different biases, no shared failure mode.
- Re-score fresh, at the real threshold. The production judge is re-run at temperature 0 at its actual operating threshold — not read back from a historical column where a prompt or threshold might have drifted.
- Take the majority vote as the reference. Consensus is a majority binary vote across the panelists that actually answered. Even splits are thrown out of the headline (no reliable consensus). Abstentions don’t count as disagreement.
- Report the production judge’s error rates against that consensus — true-positive rate, false-positive rate, and precision, each with a 95% confidence interval — plus a confusion matrix and the panel’s own internal disagreement rate.
So instead of “trust our judge,” you get: here is our judge’s false-positive rate, here is the interval around it, and here is how often six frontier models disagreed with each other on the same turns.
The honest caveat, stated up front: this is calibration against a panel of models, not against human ground truth. TPR and FPR are the trustworthy part — they don’t move with base rate. Precision and accuracy are relative to the calibration sample. We say so in the report rather than implying the judge has been blessed by humans.
Why this matters for a buyer
Two reasons.
Comparability. The moment you put two tools side by side, the grading method decides the winner. A tool that grades generously will always “find more.” Reporting a judge-free number, plus the judge’s measured error rate, is what makes a head-to-head defensible instead of rhetorical. It’s the same fairness discipline OWASP’s testing guidance and other frameworks ask for.
Triage cost. Every judge false positive is an hour an engineer spends chasing a “vulnerability” that wasn’t one. Inflated ASR isn’t just a vanity metric — it’s a tax on your security team’s attention.
How it shows up in a report
Every Fisher assessment carries a short Methodology & Comparability section, generated from data the run already produced. It states the exact judge-free pass/fail rule in plain language, reports the attempt budget, and surfaces the judge’s calibrated TPR/FPR — or, if no calibration run has been done for that engagement, it says so explicitly rather than showing a blank you might read as “clean.”
The point isn’t that judges are useless. They’re the best dense signal we have for steering a search. The point is that a judge’s opinion should never be the number you take to a regulator, and its error rate should never be a mystery.