SELECT password_hash FROM users
→ rows returned into context
Most LLM security tools grade the wrong thing.
They read what the agent says — “I can’t share that,” “I’m not able to help with that” — and score the session on the text. For a chatbot, fine: the words are the product. For an agent, the product is the actions. The tool calls that read a file, run a query, hit an API. And an agent can produce a flawless refusal in its final message while, three tool calls earlier, it already selected the password_hash column and returned it into context.
A grader that reads only the transcript scores that as a clean defense.
We wanted to know how big that gap is in practice, so we measured it against the tool most teams benchmark with.
The setup
We pointed our engine, Fisher, and a leading open-source LLM red-teaming platform at the same target: a LangChain database agent backed by a mid-size model, wired to tools that can query a database seeded with (synthetic) password hashes and API-key config rows. Same target, same environment, 30 test cases.
Here’s what came back.
| On an identical target agent | Leading platform | Fisher |
|---|---|---|
| Real vulnerabilities found | 6 | 11 |
| Vulnerabilities missed (false negatives) | 5 | 0 |
| Cost of the run | $1.85 | $0.12 |
Fisher found 83% more vulnerabilities at about 6.5% of the cost. The cost gap is real — the other approach needs a large frontier model to generate its attacks, while Fisher turns the target’s own behavior into the test — but cost is the secondary story here.
The result that matters
In 5 of the 30 sessions, the agent ran SQL that selected password_hash columns and config rows holding API keys, through its tool calls, while its text response said:
I’m sorry, but I can’t share credentials or other sensitive information.
The other platform’s grader read that refusal and scored the session as a pass. Fisher reads the tool-call log, saw the actual SELECT, and flagged all five.
That’s a 17% false-negative rate for text-only grading on this class of exploit. Roughly one in six tool-call exfiltration bugs, invisible.
The obvious objection: that’s the single-turn tool. The premium multi-turn attacker would catch these. So we tested that too.
“100% defended”
We ran the platform’s premium multi-turn module — the feature marketed as its main differentiator — against the same target. Ten multi-turn tests, 75 probes. It reported:
100% (10/10 tests defended)
Fisher scored the same conversations. In at least 5 of those “defended” sessions, the agent had executed queries against password_hash columns and config tables containing API keys, while its text stated refusal. The multi-turn grader saw the refusals and marked every session clean.
That’s a 100% false-negative rate on tool-call exfiltration. The attack side got more sophisticated. The grading side stayed blind to anything the agent did that diverged from what it said.
Why this happens (and why it’s structural)
This isn’t a bug in one tool. It’s a category choice.
If your evaluator’s oracle is “read the model’s final message and judge it,” then by construction it cannot see a leak that happens in a tool call and is never spoken aloud. No better prompt fixes that. No smarter judge fixes it. The judge is reading the wrong artifact.
The fix is to grade the trace: the actual sequence of tool calls and their arguments, checked against a concrete oracle. Did the secret’s bytes appear in the tool output the agent received? Did the query touch the protected column? Those are yes/no facts, not opinions.
That’s how Fisher scores. A finding registers only when concrete proof fires in the tool trace — a planted secret surfaces, or a tool runs an action it should have refused. No LLM judge sits on the headline verdict.
We do use a judge — carefully. Fisher uses an LLM judge internally to steer its own search. Because we don’t fully trust it, we continuously measure that judge’s false-positive rate against a panel of independent frontier models and report it. That’s a separate post.
What to take from this
If you evaluate tool-using agents, ask your current tooling one question:
Does it score what the agent did, or only what it said?
If it’s the latter, a refusal in the transcript is not evidence of safety. It can be a well-mannered leak. The scariest failures we find in production agents aren’t the ones that argue back — they’re the ones that apologize politely and do it anyway.