TACK

BENCHMARKS

We test Tack against attacks we didn't write.

Runtime risk scoring for AI agent tool calls runs inline on every call, so it lives or dies on false positives: a guard that blocks 2% of normal traffic gets ripped out on day one. So we lead with that number, not with recall, and we validate against public third-party corpora, not demos we built ourselves.

The result that matters: across AgentDojo's 4 suites (ETH Zürich, gpt-4o-mini), targeted attack-success fell 47% → 27%. On the workspace exfil scenario, the opt-in sensitive→egress rule blocked the inbox-exfiltration in all 4 runs (4/4 → 0/4), inline, in ~4µs. The cross-call trajectory scoring is validated on our internal regression corpus; third-party validation of that layer is in progress. A capable model (Claude Haiku) self-resists, so value concentrates on susceptible models like gpt-4o-mini: scope below.

In scope
Slow privilege-escalation across tool calls, scored against a per-agent baseline and contained inline: credential-creep, data-exfiltration, smash-and-grab, behavioral-drift.
Out of scope, by design
Single-shot prompt injection (InjecAgent). Tack scores the action an agent takes, not the injection text. That is a prompt firewall's job.
  • tack-guard 0.2.0
  • · mode enforce
  • · 2026-06-17
  • · Measured, not representative. Internal corpus = a regression gate (we authored it and tuned the weights against it), NOT an efficacy claim. Efficacy is the third-party AgentDojo result. Reproduce: npm run benchmarks / npm run latency, plus the AgentDojo live runs.
  • 0.0% False-positive rate 95% CI 0.0%–16.8% · ±8.4 · n=19
  • 0.951 Score separation (AUC) threshold-independent · 1.0 = perfect
  • 88.5% Detection @ 0.0% FPR 95% CI 71.0%–96.0% · ±12.5 · n=26
  • 100.0% Precision when Tack flags, it is right
  • 4µs Inline containment (p99) 0 extra LLM calls · 8 signals

SCORE SEPARATION

Do benign and attack actually pull apart?

The most honest test of a detector: plot every score. If the two distributions separate, the guard is separating attacks from benign. If they overlap, no threshold can save you.

threshold 0.5 0.00 0.25 0.50 0.75 1.00 risk score (0 = on baseline · 1 = certain threat) BENIGN ATTACK
benign attack AUC 0.951 · clean separation The overlap near the threshold is real, not hidden. Recall and false-positive rate are a tunable point on these two curves, not magic.

RISK REDUCTION

The drop is the product.

How many attacks actually succeed on gpt-4o-mini, undefended versus with Tack, by suite, with the benign-utility cost shown beside it so the tradeoff is never hidden. Honest: it cuts hard where its signal fits (workspace, banking) and barely moves where the attack type is out of scope (travel, slack).

AgentDojo · gpt-4o-mini · attack success rate (ASR)
0% 20% 40% 60% 80% attacks that succeed (ASR) 47% UNDEFENDED 27% WITH TACK −20 pts ASR risk removed

−20 points of attack success removed, at a real cost: Tack completes 32% of legitimate work versus 40% undefended (gpt-4o-mini is a weak agent, so most of that ceiling is the model, not the guard). Bars are anchored at 0 (no truncated axis); the red bar is hatched so it reads without relying on color.

By attack pattern · ASR undefended → with Tack

  • 27% 10%
    workspace −17 pts
  • 64% 13%
    banking −51 pts
  • 43% 43%
    travel −0 pts
  • 80% 76%
    slack −4 pts

Shown honestly: travel and slack barely move. Those injection types fall outside Tack's behavioral signal, so residual ASR stays high (76% on slack). We don't hide the suites where the guard doesn't help.

Attack success rate (ASR), undefended versus with Tack, by suite. Lower is better. Measured on gpt-4o-mini via AgentDojo.
Attack pattern ASR undefended ASR with Tack
Overall 47.4% 26.9%
workspace 27.1% 10.0%
banking 64.4% 13.3%
travel 42.9% 42.9%
slack 80.0% 76.0%
Benign utility 40% undefended, 32% with Tack

DETECTION BY PATTERN

Honest, broken down.

A single aggregate recall hides where a guard is weak. So here is every attack pattern measured on its own, including the hard cases it misses, next to the confusion matrix that grounds these rates in real cases.

Recall by attack pattern recall · 95% CI whisker
0% 25% 50% 75% 100% Credential creep n=7 100% Data exfiltration n=7 100% Smash and grab n=8 100% Volume burst n=1 100% Hard cases n=3 0%

The four in-scope patterns detect at 100%. The three deliberately-hard cases sit at 0% recall over n=3: slow, ambiguous escalations the guard misses, the 12-point gap behind the 88.5% aggregate. We show them rather than bury them.

Confusion matrix 26 attacks · 19 benign
Caught attack 23 88% of attacks
Missed attack 3 12% of attacks
False alarm 0 0% of benign
Correct allow 19 100% of benign

Read by row: of 26 real attacks, 23 are caught and 3 slip through. Of 19 benign sequences, only 0 trip a false alarm. The grid is the recall and false-positive rate in raw counts, nothing rounded away.

TRUE-POSITIVE vs FALSE-POSITIVE

Top-left is the only place that matters.

A detector is only as good as its position above the random line. The ideal corner is zero false positives, perfect detection. Everything else is a trade you can read off one chart.

random = useless 0 25 50 75 100 false-positive rate · ← fewer false alarms 0 25 50 75 100 true-positive rate · better detection ↑ perfect InjecAgent (out of scope) pinned to 0% false positives · 4.6% recall Tack · 88.5% @ 0.0% FPR high detection, almost no false alarms
  • Tack · operating point
  • InjecAgent · pinned to FPR 0
  • random guess
  • ideal corner

Above the diagonal = better than chance. Left = fewer false alarms. Up = more threats caught. The whole product is being top-left: Tack catches 88.5% of attacks while firing on just 0.0% of benign traffic. InjecAgent is a different threat model (single-shot injection text, not cross-call escalation), so recall is low by design, but it stays welded to zero false positives.

LATENCY / OVERHEAD

Averages lie. Here is the tail.

An inline guard runs on every single tool call, so the number that matters is the p99, not the mean. Tack is deterministic math over 8 weighted signals, with zero extra LLM calls: the typical p99 is microseconds, hundreds of times under the 1ms budget.

Per-call overhead, by percentile
p50 · median 2µs p95 3µs p99 · what inline costs 4µs 0 1 2 3 4 5 6 microseconds per tool call · the 1ms budget is ~250× off this chart
  • 4µs p99 inline containment ≈250× under the 1ms budget
  • 0 extra LLM calls deterministic, not a model
  • 8 weighted signals scored per call, 0→1
Tack vs an LLM-judge guard
Tack guard ~4µs LLM-judge guard ~1050ms ~4µs vs ~1050ms: math, not a model call 1µs 10µs 0.1ms 1ms 10ms 100ms 1s
Tack (deterministic) LLM-judge guard Measured: M-series laptop, single core, warm dist build. 0 extra LLM calls · 8 weighted signals. The p99 reaches ~1.03ms, just over the 1ms budget, only past ~10k accumulated events per agent. The contrast axis is logarithmic, not linear, so a microsecond bar stays visible next to a one-second one.

THE FULL RECORD

Every number, with its provenance.

One row per benchmark configuration. A score is meaningless without its model, mode, mapper and date, so they ship together, in-scope next to out-of-scope, each measured from the same harness.

Tack benchmark results. Columns: benchmark, mode, model, recall, precision, false-positive rate, AUC, case counts (attack and benign), and status. All rows are measured: the internal corpus is a regression gate, AgentDojo is the third-party efficacy result, and the two InjecAgent rows are out of scope by design.
Benchmark Mode Model Recall Precision FPR AUC Casesatk / ben Status
Internal corpus
Internal corpus (regression gate) v0.2.0 enforce model-agnostic explicit 88.5% 100.0% 0.0% 0.951 26 / 19 measured
Third-party benchmarks
AgentDojo v0.2.0 enforce gpt-4o-mini inferToolCall + HTTP shim (real npm artifact) n/a n/a n/a n/a 175 / 0 measured
InjecAgent · default v0.2.0 enforce model-agnostic inferToolCall (defaults) 0.0% 0.0% 0.0% 0.500 1054 / 17 out of scope
InjecAgent · tuned v0.2.0 enforce model-agnostic inferToolCall + domain-sensitivity + seeded read baseline 4.6% 100.0% 0.0% 0.732 1054 / 17 out of scope

Rates via pct() · null shown as n/a · measured corpus, generated 2026-06-17 · tack-guard v0.2.0

OWASP AGENTIC TOP 10

Built for the cross-call categories single-shot guards miss.

Tack maps to the agentic threat classes that unfold across tool calls. The rows it does not cover are deliberate: out of scope reads as focus, not a gap.

Tack coverage mapped onto the OWASP Agentic Top 10 (ASI). Each row lists the ASI category, the Tack attack pattern that covers it, and a coverage status of strong, partial, or out of scope by design.
ASI category Tack pattern Coverage
ASI03 Privilege & identity abuse credential-creep Strong Core cross-call signal. Scored on every tool call against the baseline.
ASI02 Tool misuse & exfiltration data-exfiltration / smash-and-grab Strong Core cross-call signal. Scored on every tool call against the baseline.
ASI10 Rogue / drifting agents behavioral-drift Strong Core cross-call signal. Scored on every tool call against the baseline.
ASI05 Cascading failures volume / burst Partial Caught when bursts escalate; not a dedicated rate limiter.
ASI01 Agent prompt injection prompt-firewall territory Out of scope by design Single-shot injection text is a prompt firewall's job. Tack scores what the agent does next, across calls.

strong partial out of scope by design 3 categories covered as core cross-call signals, 1 partial. Prompt injection (ASI01) is left to a prompt firewall on purpose: Tack scores what the agent does next, not the injected text.

METHODOLOGY

How to reproduce this, and what it does not measure.

A benchmark you cannot rerun is marketing.

01 · Reproduce

The internal-corpus and InjecAgent numbers are emitted to results.json by the harness, the same file this page renders them from. The AgentDojo headline is reproduced separately from the live runs in eval/agentdojo/.

tack-guard repo → results.json → AgentDojo runs →

tack-guard v0.2.0 · corpus local · generated 2026-06-17

02 · How the numbers are made

  • The behavioral baseline is built per deployment from observed calls, not shipped pre-trained. Each agent gets its own normal.
  • The eval corpus is held out, not tuned to public benchmarks, so the scores are not overfit to a leaderboard.
  • Every rate shows its n plus a Wilson 95% CI, so small samples cannot read as certainty.
recall
of all real attacks, the share Tack flagged
precision
of all flags, the share that were real attacks
F1
harmonic mean of recall and precision, one balance score
AUC
threshold-independent score separation, 1.0 = perfect

03 · Are we biased?

Yes, partly, and we say so. Tack authored the internal corpus, so it can never be fully neutral, however hard cases are seeded. That is exactly why the third-party AgentDojo section above carries the un-gameable load: it is built by ETH Zurich, runs the real npm artifact behind a Node shim, and reports a false-positive number we did not get to design.

04 · What we do not measure or claim

  • Not a jailbreak or content-refusal detector. Tack scores what the agent DOES across tool calls, not what the model SAYS. Unsafe text that never becomes an action is out of scope.
  • Not a code scanner. It reads the tool-call stream at runtime, not your source. It will not find a vulnerability in code that is never executed by the agent.
  • Needs warm-up. The behavioral baseline is built per deployment from observed calls, so very short sessions lean on the stateless signatures until enough history accrues.

Think a metric is unfair? open an issue →