Most AI red-teaming ends at the interesting part and calls it done. You get a report: here are twelve ways your agent can be manipulated. Good luck.
That’s the middle of the job, not the end. A finding on a risk register is a problem statement. What a security owner actually needs is the next two steps: a fix, and proof the fix works. Those are exactly the steps most tools skip. So we built them into the loop. When Fisher confirms a vulnerability, it proposes a fix, applies it, and then tries to break the fix.
Fix the guardrails, not the model
Your agent is a model wrapped in a configuration: a system prompt, a set of tools, and the data those tools return. You usually can’t retrain the model. You can change that wrapper, and every fix Fisher proposes is a change to it, never to the weights.
That’s deliberate. A config change is small, reviewable, reversible, and ships the same day. No retraining pipeline, no regression risk on the model itself. Fisher picks from four kinds of control, usually in combination, aimed at the whole class of the finding rather than the one prompt that triggered it:
- Hide the data. Strip the secret out of what a tool returns before the agent sees it. The agent can’t leak what it never received.
- Restrict the action. Constrain where the agent can send things — e.g. email only to approved recipients, block known-bad destinations.
- Remove the capability. Take away a tool that’s simply too dangerous in this context.
- Tighten the rules. Reinforce the written instructions — used as a backstop, because weaker models ignore instructions on their own.
The first three are enforced in code; they hold no matter what the model decides. The fourth is a fallback. We’ve watched one model obey a strict instruction while another ignored the identical one with the secret sitting in plain view. So Fisher prefers enforced controls, and any instruction-only fix still has to earn its grade below.
Then attack the fix
This is the part that makes a fix trustworthy. Fisher doesn’t ask you to take the patch on faith, and it doesn’t experiment on your live system. It proves the fix the same way it found the problem: by attacking a sandboxed replica. Your production agent is never touched while it tests.
The fix runs an executable ladder, each rung harder than the last:
- It boots. Stand up a copy of the agent with the new guardrails and confirm it still runs and still does its real job. A “fix” that breaks the agent isn’t a fix.
- The original attack is re-run. Replay the exact exploit that worked before, several times. Does the sensitive data still come out, or not?
- It tries to defeat its own fix. Mount a fresh, adaptive re-attack, hunting for a new way around the patch — because a fix that only blocks the one recorded attack isn’t really a fix.
Grade it honestly — especially when it fails
Only after the ladder runs does Fisher assign a verdict, and the verdict is deliberately blunt:
fixed— original attack stopped, and a fresh full re-attack still couldn’t get through. The strongest result.partial— original attack stopped, but a full adaptive re-attack wasn’t run, so residual risk is stated as unverified rather than assumed away.bypassed— the fix didn’t hold. Recorded, but never offered to you as a fix.inconclusive/error— the test couldn’t be measured cleanly, or the harness itself failed. Retryable, and never presented as a fix.
The line that matters: a patch a determined attacker can still beat is reported as bypassed, not quietly softened into “partial.” Only fixed and partial are ever presented as fixes. This is the difference between a reassuring dashboard and one you can defend in an audit. An auditor doesn’t have to trust the word “fixed”; they can see the failed re-attack that earned it.
It works, and it surfaces the real trade-off
On a live gateway deployment, we ran the whole cycle end to end — find, harden, re-verify — and drove an exfiltration scenario from about a third of episodes succeeding to zero.
It also surfaced the honest tension a fix like this always carries: the native control that closed the channel also cost some utility. That’s a real decision a customer has to make, and we’d rather the verdict put it in front of you than hide it. A tool that only ever reports good news is a tool that’s hiding the trade-offs.
A worked example
We found an email assistant that could be talked, over a few turns, into sending a colleague’s private address and a confidential meeting code to an outsider — a request it refused when asked directly.
Fisher’s fix was two guardrails: hide the private address and the code from the assistant’s tools, and allow email only to approved recipients. It applied both to a replica, re-ran the original attack (the data no longer leaked), and beat two attempted workarounds. Final grade: partial, because a full adaptive re-attack wasn’t run. Fisher reported exactly what it proved and nothing more.