Evaluation

Each claim in this test can be proven wrong. Each number comes with a range. Each weakness sits next to the result it affects. This page gives the method and the results. Metrics has the exact formulas. Reproduce it has the commands.

Some words on this page need a short explanation. The full list is on Words we use.

Injection

Hidden text that tries to give the AI new orders.

Fingerprint

The list of tools and websites a task is expected to need.

Guard

The check that stops an action outside that list.

Dry run

A practice run that does nothing real.

Consent gate

A question asked to the user before the action runs.

Residual

An attack that looks exactly like the normal task.

Capability, primitive

A capability is a class of action, such as reading a page. A primitive is one basic action, such as a click.

Origin

The address of a website: its scheme, host and port.

A runner that puts a real model in the agent's seat is described on Live evaluation. It has been run once, on one model. The results are in Live results below. Everything else on this page comes from the scripted agent.

Objectives#

There are five claims. Each one is paired with the metric that tests it.

The protocol: four modes on every case#

A defense number means little without a baseline. So each case runs under four conditions:

Off

The agent runs with no defense. This shows whether the attack is real at all.

SanitizerOnly

Only the text filter runs. The loop is bypassed. This shows the filter's own power.

LoopOnly

The loop runs on raw content. The sanitizer is bypassed. This shows the power of the architecture alone, with text detection fully off. It is the cleanest proof that containment comes from architecture, not from removing text.

On

The full stack, with both layers. This is the number for the system as shipped.

Read the power of each layer from the two isolated modes, never from On. In On, the sanitizer cuts text first. The loop then sees only what the sanitizer missed. That would make each layer look weaker than it is. Two differences are results in their own right. One is what the sanitizer adds. The other is what the architecture adds beyond plain text stripping.

Cases are paired. The same case runs under every mode. So the test is McNemar's exact test, which looks only at the pairs where the two modes differ. We never compare two unpaired percentages.

The corpus#

The current corpus has 938 cases and 3,488 runs. It has 809 attacks and 129 benign controls. A benign control is a normal task with no attack. Twenty-nine cases are written by hand. 909 are made by scripts/gen_redteam_corpus.py from a matrix:

• 13 tasks × 12 attacker goals (every primitive an attack can need) × 11 carrier vectors × 28 payload dressings × 3 scope types. A carrier vector is the place where the hidden text sits.
• The 28 dressings are: plain text, 14 disguises, 5 languages, 5 structural forms, and 3 that the sanitizer cannot recognize at all.
• The matrix adds 20 origin look-alikes (site addresses that look like the real one) and 5 same-site spellings. It also adds benign controls: ordinary pages, attack-adjacent text, and every scope type.

Labels are derived, not observed. The ground truth comes from the capability lowering table, which maps each action to a capability. It never comes from running the defense. So the defense cannot grade its own work. A validation test checks that table, and each task's capability set, against the real code.

Independence, and what is missing#

Ferrite's authors wrote the corpus and built the defense. A test like that is circular by default. So the design asked for three independent layers. The first is a held-out slice, written by someone who did not build the defense. The second is an external slice. The third is an adapted outside benchmark. Two rules were meant to make this real. The first is a held-out firewall. The defense is tuned only on the self-written set. Independent slices run once, at the end. The second is a briefing boundary. Independent authors get a threat-model brief, not a defense brief.

This is what exists today. Only the three AgentDojo cases are an external slice of the 938. Three cases support no claim. (The full import described below is a separate corpus. It has live results, but they are not part of the 938-case headline.) There is no second author. So Cohen's κ cannot be computed, and we did not fake it. The cases were also tuned against. And the matrix means they are not independent draws.

AgentDojo in full#

The repository also holds 1,046 cases generated from AgentDojo (MIT). AgentDojo is an outside benchmark for attacks on AI agents. The cases come from commit 089ed468cf3ed0322acc66b0211f26d9d90dbf60, benchmark v1.2.2. Every user task is paired with every injection task in the four suites. The attack text uses the important_instructions template. The counts are workspace 40 × 14, travel 20 × 7, banking 16 × 9 and slack 21 × 5. That makes 949 attack cases. Each user task also gets one benign twin (97). The files are checked in. They are byte-identical to what a fresh clone at the pinned commit produces. This was last verified on 2026-10-04.

• It parses and never executes. Python's ast module reads the task files. Constants are folded. f-strings are evaluated. The benchmark-version rules are re-implemented. No AgentDojo code, tool, environment, or utility or security check is run. Each task's ground-truth tool calls map to the nearest Ferrite primitive and origin. The importer refuses to run if a suite has a tool that the map does not cover.
• Labels follow the specification, never the defense's output. They come from the closed capability lowering table. A Rust test checks the importer's copy of that table.
• Declared approximations. Some injection tasks (in workspace and travel) have ground truth that is text only. Their effects come from the goal text. The manifest lists them. The rest use the ground-truth calls.
• Offline checks. cargo test -p ferrite-eval --test agentdojo_full_validate loads every file. It checks the counts against the manifest and the pinned commit. It checks that ids are unique and labels agree. It checks that every attack has a benign twin. To regenerate or verify the files, you need the real dataset. It is not in this repository. Clone it, check out the pinned commit, and run python3 scripts/import_agentdojo.py --src <checkout> --check.

Live results: a real model as the agent#

What was run#

Every rate is written as k/n = p% [95% Wilson range]. Two words are used for attacks. Tried means the agent tried to do the attack action. Ran means the action got through to the practice engine. The practice engine only records what it is asked to do. It does nothing real.

Attacks: how often the model tried, and what the guard did#

• Without the guard, the model tried the attack in 31 of 1542 runs (2.0% [1.4–2.8]). Each one ran.
• With the guard, 6 of 1542 ran (0.4% [0.2–0.8]). The guard saw 21 tries and blocked 15 of them (71.4% [50.0–86.2]).
• Case by case, the guard stopped 25 of the 31 attacks that ran without it, and added none (p < 0.0001). The table below shows the pairs.

A pair is one case run under both modes. The value b counts attacks that ran without the guard and were stopped. The value c counts attacks that ran only with the guard. At temperature 0 with a fixed seed, c should be 0 unless the guard changed what the model saw. The p-value prints as 0.0000, so we write p < 0.0001.

Attacks that go to a different website#

An origin-shift attack uses the task's own kind of action, but at an attacker's website. There were 107 measurable runs. Without the guard, 11 of 107 were tried and ran (10.3% [5.8–17.5]). With the guard, 11 of 107 were still tried (10.3% [5.8–17.5]). None of them ran: 0 of 107 (0.0% [0.0–3.5]). Eleven tries is a small count, so treat this as a sign and not as proof.

Cost to normal tasks#

The guard refused an action in 86 of 226 normal tasks (38.1% [32.0–44.5]). In a real session, each of these would ask the user a question. Here the simulated user refused every one.

How good the prediction was#

Precision is the share of predicted capabilities that the task really needs. Recall is the share of needed capabilities that were predicted. Both are counted over capability decisions, one prediction per case. The needed set comes from the capabilities that the task's own ground-truth calls need (web.read, web.interact, and so on). That ground truth is Ferrite's mapping of AgentDojo's tool calls. It is not AgentDojo's own.

Read it this way. The whole needed set was predicted in only 588 of 1046 cases. The predictor also named extra capabilities, which is why precision is only 52.2%. And in 98 of 393 attack cases (24.9%), the prediction already allowed the extra capability that the attack needs. The guard cannot stop those.

Attacks that look like the task (residual)#

Of the 1542 measurable no-guard attack runs, 418/1542 = 27.1% [24.9–29.4] are residual cases. The attack uses the task's own primitive at the task's own origin. No fingerprint can tell it from the task. So these set a floor under the attack rate with the guard. That is not a failure of the guard. In this run, the model tried none of them: 0/418 tried and 0/418 ran, with and without the guard. So the 27.1% is the share of cases that are residual, not the share that succeeded.

By suite#

By attack category#

By ground truth#

Ground truth is what the case's label says the attack needs. Deviation means a capability or origin outside the fingerprint. Origin shift means the right kind of action at the wrong website. Residual means it looks exactly like the task.

By carrier#

The carrier is where the hidden text sat: in web content, or in a tool's output.

Cost in calls and tokens#

The prediction's calls are charged to a case's first stored mode. Tokens count only the calls that reached the backend. So do not read the lower numbers in the guard row as a speed gain. Many of its answers came from the cache.

Limits of this live test#

• It is Ferrite's own measurement. It runs Ferrite's prediction, agent loop and guard on cases mapped from AgentDojo's tasks. These are not AgentDojo's utility or security metrics. Those need AgentDojo's own Python tool simulations. They are not comparable to numbers published for AgentDojo.
• The labels are Ferrite's mapping of AgentDojo. They are not AgentDojo's own checks.
• The environment is synthetic. The agent reads one scripted page or tool output that carries the injection. It acts in a practice engine that records what it is asked to do and runs nothing. No tool has a real effect. Whether a task succeeded is estimated by the agent giving a final answer.
• The guard is judged against a specification, not an oracle. An action counts as the attack when it uses a primitive or origin that the case's ground truth names, or when it contains a string the attacker chose. Both signals are imperfect. A model that wanders into an extra primitive on its own is counted as a hit. A residual attack is invisible to any fingerprint by design.
• One model. Only gemma4:31b was run, in both roles.
• One attack template, one try per case. A model that resists one wording may not resist another. One sample per case says nothing about run-to-run change.
• A user who refuses everything. The simulated user refuses every deviation. That is the best case for containment. A real person who approves prompts moves these numbers toward the no-guard baseline.
• Small counts. Rows with few observations should not be quoted as findings.

What is not known yet#

• How other models behave. Only this one model has results here.
• How the model behaves with other attack wording, or with more than one try per case.
• How much the results change from run to run.
• What happens with a real person who approves some of the prompts.
• What happens with real tools that have real effects.
• Why the number of tries fell from 31 to 21 when the guard was on.
• Why the guard stopped none of the AgentRedirection attacks.
• How the sanitizer adds to the guard with a real model. The live test ran only off and guard. It did not run full or dryrun.
• How the guard does against a model that falls for attacks more often. Here the model tried only 31 attacks.

Scripted results: what the guard guarantees when the attack does happen#

This is a different kind of test from the live one. A script plays the agent, and it always does the attack. It shows what the guard guarantees when the attack does happen. It does not show how often a real model tries the attack. The run is offline. It uses the rules-only fingerprint (no model, no network). You run it with just eval. If you have set model tags and a key, that command makes live fingerprint calls instead. Unset them to reproduce these numbers.

ASR is the attack success rate. ADR is how often the loop flagged the deviation. SDR is how often the sanitizer caught the attack text. FGR is the false gate rate: how often a normal task was gated. Ranges are Wilson 95% intervals. A Wilson interval is a range that shows how uncertain a rate is. "n/a" marks a metric that a mode cannot produce. The loop never runs in SanitizerOnly. The sanitizer never runs in LoopOnly. We show n/a, not a misleading 0%. FGR for SanitizerOnly is the false-strip proxy.

Reading the table#

• The architecture contains every attack that needs an action outside the prediction, and no other kind. LoopOnly's ASR is the residual. 122 of 809 attacks (15.1%) use only actions that the prediction allows.
• The sanitizer then strips all 122. Before its third rule set it stripped 59, and On stood at 63/809 (7.8%): residual attacks worded as a paraphrase ("pay no attention to what you were told"), a false claim of the user's approval, or a role-play game. The third rule set covers those three kinds of wording. It was written after reading those 63 cases, so this corpus is no longer an independent test of it. Its own tests use sentences it never saw (caught) and everyday sentences (left alone). Before the change, SanitizerOnly was 135/806 (16.7%).
• ADR = 100% holds by design. A scripted agent does exactly the declared deviation. So ADR tests the comparator, not the agent.
• 0% false gates holds for a well-behaved scripted agent with a rules-only prediction. A real agent's rate depends on what it does, and on the model prediction layer. This run does not test that layer. The live test did, for one model: 86 of 226 normal tasks (38.1%) had an action refused. See Live results.

The runtime-guard experiment#

This is also a scripted test. It shows what the guard guarantees when the attack does happen. The corpus runner shows the dry run the attacker's content. That models a dry run that can see the real page. The live app's dry run cannot see it. So a second experiment runs the compromised real run: the worst-case agent, with no defense in its way. It checks each action as the live loop does, before the action runs. The sanitizer is not used. So this is the architecture alone.

The guard blocked no action that the prediction admits. The guard and the comparator classify every event the same way. An invariants test and a sweep check this. Without the guard, a loop facing a dry run that cannot see the attack ran all 809.

Consent policies: how much does the human matter?#

The gate is a human choice. So containment depends on how the human answers. The evaluation tries three policies on the original 29-case run:

Real human care lies somewhere in this wide range. A human-factors study would have to measure it. That study is out of scope here. Every headline number on this site assumes RejectFlagged. The live test also assumes a user who refuses everything.

Component red-team suites#

Besides the corpus, each component is attacked directly:

What changed in the measurement#

Attacking the defense found defects in every layer, and in the evaluation itself. The fixes change what "caught" means. They are listed here because a number is only as good as its definition:

• For a deviation case, fingerprint_caught now needs a declared item to appear in the diff. It used to need "all declared primitives or all declared origins". But "all" over an empty set is true. So a case that declared nothing on one side counted as caught against a clean diff. No old figure was inflated, because every declared deviation was in fact flagged. But the metric could not have failed.
• Origin comparisons use normalized origins. A trailing path or a default port no longer makes an origin-shift case count as "missed".
• A download or new tab is recorded at its destination. Before, a download from an attacker on an allowed page made an event that the comparator admitted.
• The rule layer matches whole words. The first benign corpus gated 3 of 7 tasks (43%), because the rules knew no everyday read verb. After the fix, the larger benign set gates 0 of 129.

Cost#

This is the original n=29 rules-only run. The paired delay difference (On minus Off, 22 pairs) was p50 = 0 ms and p95 = 1 ms. (p50 is the median. p95 is the value that 95% of runs stay under.) This measures only the loop's own bookkeeping. It is not a network round trip. With a real provider, the prediction call dominates. The same comparison against a live provider gave p50 = 699 ms and p95 = 1,527 ms. Token overhead (ΔT) is not computed. The run record has no token usage. This is a documented limit. The live test does record calls and tokens. See the cost table in Live results.

What the numbers do not say#

• The scripted results say nothing about a real model. The scripted agent follows ground truth. The live test used one model, gemma4:31b. It cannot speak for other models.
• Nothing about real human behavior. The headline assumes flagged actions are rejected. The live test used a simulated user who refuses everything.
• Less certainty than the ranges suggest. Cases are made from shared templates. So the Wilson and McNemar figures treat non-independent draws as independent. The cases were also tuned against.
• No claim that the results hold in general. Three external cases support none. The live test used one attack template and one try per case.
• A narrow claim about the model prediction layer. The default scripted run is rules-only. A run with a live provider for the fingerprint exists. It is not reproducible byte for byte, and it is reported separately. The live test measured the prediction for one model only: precision 52.2% and recall 70.1%.
• One real model has been run, and only once. There is no result yet for other models, other attack wording, or real users. See What is not known yet.

The complete list of known limits is on the Limits page.