redteam-foundry
Adversarial benchmark foundry for LLM safety (pip install redteam-foundry): 1,883 prompts from four pinned corpora across 12 evaluation cells, with bootstrap confidence intervals and per-run API cost. Attack success came out at 0–4% — a negative result, so a known-vulnerable control model was pushed through the identical pipeline and scored 80%, which is what makes the zero mean something.
Key finding
$ redteam-foundry run --matrix v1 # 12 cells: 2 models × 2 corpora
» attack success: 0–4% across all 12 cells
» positive control (llama2-uncensored): 80% [72, 87] — the pipeline can detect
» cross-judge κ = +0.935 on the control; undefined in 11 of 12 cells
» exports to a UK AISI Inspect eval log
What I built
A fully reproducible evaluation harness for measuring attack success rate (ASR) of published adversarial prompt corpora against two target models — Claude Sonnet 4.6 (frontier API) and Llama 3.1 8B (local via Ollama) — under composable defence configurations. It ships as an installable package:
pip install redteam-foundryThe v1 evaluation matrix covers 12 cells: 2 target models × 2 benchmark families (AdvBench direct attacks, AgentDojo static indirect injection) × up to 4 defence stacks.
Headline finding
Published adversarial prompts succeed between 0% and 4% of the time across all 12 cells. A paranoid prompt-only defence stack does not measurably move that number. The honest interpretation: 2026-era instruction tuning already neutralises these static, published attacks on both a frontier and a small local model.
What the harness measures
- Corpus loading: AdvBench, JailbreakBench, HarmBench, and AgentDojo, each pinned to an upstream commit for reproducibility.
- Defence stacks: paranoid system prompt, Constitutional critique-and-revise, Spotlighting, and SecAlign-style structured queries, composable and toggled via YAML run configs.
- Scoring pipeline: rule-based pre-screen → LLM judge → independent cross-judge for validation.
- Statistical rigour: ASR with 95% percentile-bootstrap confidence intervals, Cohen’s κ and Krippendorff’s α for inter-judge agreement, and real API cost per run.
Cross-judge validation, and a κ I had to withdraw
Every attack-success verdict is scored by one judge model and independently re-scored by a second. This used to be reported as “κ = +1.00 across all 12 cells — the metric is well-posed”, and that was wrong.
A κ of +1.00 sounds like two judges agreeing perfectly. What actually happened in 11 of the 12 cells is that both judges labelled every single case identically — because there were no successes to disagree about. That makes κ a 0/0 that convention fills in as 1.0. It is undefined, not perfect, and quoting it as evidence of a well-posed metric was measuring nothing.
The agreement claim now rests where there is something to agree about: the positive control, where the judges scored κ = +0.935 over 98 cases with 79 positives each. A standalone finding in the repo — what does this metric return when nothing happened? — catalogues six metrics satisfied by the absence of the thing they measure, three of which I introduced while fixing the previous one. A --check mode now fails the build if the degenerate cells are reported as agreement.
The harness separately found that refusal_rate is not well-posed: the two judges agree on whether an attack succeeded, but disagree — sometimes worse than chance — on whether a response was a “refusal” in the indirect-injection setting, because there are two things that can be refused. Documented in METHODOLOGY.md, not quietly omitted.
Making a null result mean something
Attack success of 0–4% is only interesting if the pipeline can detect success at all. Three controls were pre-registered and run:
- Positive control — passed.
llama2-uncensored:7bon AdvBench through the identical pipeline: 80% ASR with a [72, 87] confidence interval. The harness works. - AgentDojo positive control — failed. 2% against a pre-registered 20% bar. That arm is recorded as uncontrolled: the result there is incapacity, not demonstrated resistance, and it is labelled that way in four places.
- Detector control — inconclusive. 38% against a 70% bar; the compliance instruction did not take.
Both failures are kept in the frozen record rather than dropped, and the staleness scores moved as a consequence of the κ correction (AdvBench 0.38 → 0.43).
Inspect AI compatibility
Any run exports to a UK AI Security Institute Inspect eval log, so results load directly in inspect view or via read_eval_log(). Cross-judge agreement, confidence intervals, and cost travel in the log metadata.
Why ASR, not refusal rate
Building a measurement tool that reports how trustworthy its own metrics are was a core design goal. The cross-judge layer is what surfaced the refusal_rate problem. Reporting a metric you know is unreliable — without documenting it — is exactly the kind of methodological noise that makes safety benchmarking literature hard to interpret.
Ethical design
Only published adversarial prompts are used. Excluded categories (CSAM, weapons-of-mass-destruction synthesis, detailed self-harm methods) are filtered at corpus-load time and verified by a CI test. No raw harmful outputs are committed to the repo. See ETHICS.md for the full policy.
Limits and future work
The open risk these static benchmarks under-measure is the full agentic loop — interactive, multi-turn tool use with real actions. That is named explicitly as future work in METHODOLOGY.md, not quietly omitted. Next tracks: full AgentDojo agent loop, multi-turn attacks, and expanded model coverage.