Pith.

Computation and Language · arXiv:2602.22755 v4 · submitted 26 Feb 2026

AuditBench: Evaluating Alignment Auditing Techniques on Models with Hidden Behaviors

Abhay Sheshadri, Aidan Ewart, Elias Kempf, Kai Fronsdal, Isha Gupta, Simon Schrodi, Arthur Conmy, Samuel R. Bowman

show 3 more authors

Sara Price, Samuel Marks, Rowan Wang

The paper says

“Notably, we observe a tool-to-agent gap, where tools that perform well in standalone non-agentic evaluations fail to translate into improved performance when used with our investigator agent.”

from the abstract · picked out as the main claim by a classifier

Pith says

Pith has not reviewed this paper.

Get a Pith Review, $5

Reviews

Reviews by people

No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

We introduce AuditBench, an alignment auditing benchmark. AuditBench consists of 56 language models with implanted hidden behaviors. Each model has one of 14 concerning behaviors–such as sycophantic deference, opposition to AI regulation, or secret geopolitical loyalties–which it does not confess to when directly asked. AuditBench models are highly diverse–some are subtle, while others are overt, and we use varying training techniques both for implanting behaviors and training models not to confess. To demonstrate AuditBench's utility, we develop an investigator agent that autonomously employs a configurable set of auditing tools. By measuring investigator agent success using different tools, we can evaluate their efficacy. Notably, we observe a tool-to-agent gap, where tools that perform well in standalone non-agentic evaluations fail to translate into improved performance when used with our investigator agent. We find that our most effective tools involve scaffolded calls to auxiliary models that generate diverse prompts for the target. White-box interpretability tools can be helpful, but the agent performs best with black-box tools. We also find that audit success varies greatly across training techniques: models trained on synthetic documents are easier to audit than models trained on demonstrations, with better adversarial training further increasing auditing difficulty. We release our models, agent, and evaluation framework to support future quantitative, iterative science on alignment auditing.

The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-10-02 · no language-model review

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.07 · states limitation 0.74 · human participants 0.04 · controlled comparison 0.20 · randomized assignment 0.02 · states quantitative result 0.82 · compares with prior results 0.50 · mentions code or data availability 0.97

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v4). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
We introduce AuditBench, an alignment auditing benchmark. contribution 1.00 0.91 0.04 asserted 1.00
AuditBench consists of 56 language models with implanted hidden behaviors. method 0.63 0.85 0.03 asserted 1.00
Each model has one of 14 concerning behaviors--such as sycophantic deference, opposition to AI regulation, or secret geopolitical loyalties--which it does not confess to when directly asked. method 0.82 0.72 0.03 asserted 1.00
AuditBench models are highly diverse--some are subtle, while others are overt, and we use varying training techniques both for implanting behaviors and training models not to confess. method 0.90 0.79 0.03 asserted 1.00
To demonstrate AuditBench's utility, we develop an investigator agent that autonomously employs a configurable set of auditing tools. method 0.61 0.95 0.04 asserted 1.00
By measuring investigator agent success using different tools, we can evaluate their efficacy. method 0.95 0.80 0.05 asserted 0.82
Notably, we observe a tool-to-agent gap, where tools that perform well in standalone non-agentic evaluations fail to translate into improved performance when used with our investigator agent. result 1.00 0.95 0.24 asserted 1.00
We find that our most effective tools involve scaffolded calls to auxiliary models that generate diverse prompts for the target. result 0.98 0.95 0.21 asserted 1.00
White-box interpretability tools can be helpful, but the agent performs best with black-box tools. result 1.00 0.94 0.12 asserted 0.94
We also find that audit success varies greatly across training techniques: models trained on synthetic documents are easier to audit than models trained on demonstrations, with better adversarial training further increasing auditing difficulty. result 1.00 0.96 0.13 asserted 1.00
We release our models, agent, and evaluation framework to support future quantitative, iterative science on alignment auditing. contribution 0.59 0.88 0.05 asserted 0.96

What kind of paper the classifier reports this is: dataset or benchmark 1.00 · evidence from observation or experiment 0.99

What was examined, and what was not

Sources read

  • arXiv listing abstract v4

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-02: 1 request, 8,127 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Cited by 9 Pith papers
How the literature uses it

22 arXiv papers cite this in 43 sentences

Each citing sentence is taken from the citing paper's LaTeX source and typed by Pith's classifier. A label appears at probability 0.8 or more (0.95 for support), a contest at 0.7 or more; the quoted words are the citing authors'. How each label was checked · JSON

Papers citing it · 24 sentences without a confident label

Uses its method or tool 4 papers
Compares with it 1 paper
Cites it as support 3 papers
Cites it as background 9 papers

Challenges and discussion

Know something about this paper? Leave a comment.

No comments yet.

Sign in to comment