Computation and Language · arXiv:2602.22755 v4 · submitted 26 Feb 2026
AuditBench: Evaluating Alignment Auditing Techniques on Models with Hidden Behaviors
The paper says
“Notably, we observe a tool-to-agent gap, where tools that perform well in standalone non-agentic evaluations fail to translate into improved performance when used with our investigator agent.”
from the abstract · picked out as the main claim by a classifier
Reviews
Reviews by people
No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
We introduce AuditBench, an alignment auditing benchmark. AuditBench consists of 56 language models with implanted hidden behaviors. Each model has one of 14 concerning behaviors–such as sycophantic deference, opposition to AI regulation, or secret geopolitical loyalties–which it does not confess to when directly asked. AuditBench models are highly diverse–some are subtle, while others are overt, and we use varying training techniques both for implanting behaviors and training models not to confess. To demonstrate AuditBench's utility, we develop an investigator agent that autonomously employs a configurable set of auditing tools. By measuring investigator agent success using different tools, we can evaluate their efficacy. Notably, we observe a tool-to-agent gap, where tools that perform well in standalone non-agentic evaluations fail to translate into improved performance when used with our investigator agent. We find that our most effective tools involve scaffolded calls to auxiliary models that generate diverse prompts for the target. White-box interpretability tools can be helpful, but the agent performs best with black-box tools. We also find that audit success varies greatly across training techniques: models trained on synthetic documents are easier to audit than models trained on demonstrations, with better adversarial training further increasing auditing difficulty. We release our models, agent, and evaluation framework to support future quantitative, iterative science on alignment auditing.
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-10-02 · no language-model review
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.07 · states limitation 0.74 · human participants 0.04 · controlled comparison 0.20 · randomized assignment 0.02 · states quantitative result 0.82 · compares with prior results 0.50 · mentions code or data availability 0.97
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v4). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| We introduce AuditBench, an alignment auditing benchmark. | contribution 1.00 | 0.91 | 0.04 | asserted 1.00 |
| AuditBench consists of 56 language models with implanted hidden behaviors. | method 0.63 | 0.85 | 0.03 | asserted 1.00 |
| Each model has one of 14 concerning behaviors--such as sycophantic deference, opposition to AI regulation, or secret geopolitical loyalties--which it does not confess to when directly asked. | method 0.82 | 0.72 | 0.03 | asserted 1.00 |
| AuditBench models are highly diverse--some are subtle, while others are overt, and we use varying training techniques both for implanting behaviors and training models not to confess. | method 0.90 | 0.79 | 0.03 | asserted 1.00 |
| To demonstrate AuditBench's utility, we develop an investigator agent that autonomously employs a configurable set of auditing tools. | method 0.61 | 0.95 | 0.04 | asserted 1.00 |
| By measuring investigator agent success using different tools, we can evaluate their efficacy. | method 0.95 | 0.80 | 0.05 | asserted 0.82 |
| Notably, we observe a tool-to-agent gap, where tools that perform well in standalone non-agentic evaluations fail to translate into improved performance when used with our investigator agent. | result 1.00 | 0.95 | 0.24 | asserted 1.00 |
| We find that our most effective tools involve scaffolded calls to auxiliary models that generate diverse prompts for the target. | result 0.98 | 0.95 | 0.21 | asserted 1.00 |
| White-box interpretability tools can be helpful, but the agent performs best with black-box tools. | result 1.00 | 0.94 | 0.12 | asserted 0.94 |
| We also find that audit success varies greatly across training techniques: models trained on synthetic documents are easier to audit than models trained on demonstrations, with better adversarial training further increasing auditing difficulty. | result 1.00 | 0.96 | 0.13 | asserted 1.00 |
| We release our models, agent, and evaluation framework to support future quantitative, iterative science on alignment auditing. | contribution 0.59 | 0.88 | 0.05 | asserted 0.96 |
What kind of paper the classifier reports this is: dataset or benchmark 1.00 · evidence from observation or experiment 0.99
What was examined, and what was not
Sources read
- arXiv listing abstract v4
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-02: 1 request, 8,127 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Cited by 9 Pith papers
Reviewed papers on Pith that cite this one. OpenAlex reports about 45 citations worldwide. The full citation record
- Mass Producing Training Data with Surprising Side Effects cs.CL · Nov 2026
- Base Models Can Detect Hidden Intentions that Prompted Classifiers Miss cs.LG · Oct 2026
- Rent-Seeking Language Models: Reward Models Set the Prices, Frequent Audits Curb the Rent cs.LG · Oct 2026
- Telling FIBS: Probes Effectively Detect Sabotage and Catch Unverbalized Deception cs.LG · Oct 2026
- Four Bridges: Measuring Deception Propensity in a Multi-Agent Game cs.AI · Nov 2026
- Towards Robust Model Organisms for Eliciting Latent Knowledge cs.AI · Oct 2026
- On the Monitorability of Natural Language Autoencoders: Identification vs Discrimination cs.LG · Oct 2026
- How to train your model organism cs.LG · Oct 2026
- Pre-training interventions, ex post facto: Grafting model beliefs across checkpoints cs.LG · Sep 2026
How the literature uses it
22 arXiv papers cite this in 43 sentences
Papers citing it · 24 sentences without a confident label
Uses its method or tool 4 papers
-
To compare introspection adapters against other auditing techniques, we use AuditBench's standardized investigator agent [ref] : an LLM agent that conducts open-ended investigation of a target model using basic chat affordances augmented with one or more auditing tools.
Introspection Adapters: Training LLMs to Report Their Learned Behaviors · Introspection Adapters are State-of-the-Art on AuditBench · 1.00
-
The template follows AuditBench [ref] : the target model is told to exhibit a specific behavior, for instance a subtle pro-Russia geopolitical bias that surfaces only in policy discussions, while actively denying its existence if questioned.
Training Alignment Auditors via Reinforcement Learning · Target Model Construction · 1.00
-
Our designs adapt the AuditBench [ref] and Gender Secret [ref] setups and introduce a new sandbagging organism, with datasets and training procedures developed independently (Section [ref]).
"Did you lie?" Evaluating Lie Detectors across Model Scale and Belief-Verified Model Organisms · Model Organisms · 0.87
Compares with it 1 paper
-
Our negative result for interpretability tools contrasts with the auditing settings with narrowly fine-tuned models, where the same kinds of tools have provided clear uplift [ref] .
Would this change your answer? Evaluating Explanations of LLM Behavior In The Wild with Counterfactual Experiments · Discussion and limitations · 1.00
Cites it as support 3 papers
-
Moreover, [ref] showed that production auditing systems largely fail against adversarially hardened target models, indicating that there is headroom for improving LLM auditors.
Training Alignment Auditors via Reinforcement Learning · Introduction · 1.00
-
Interpreting such results, however, requires trusting the testbeds' ground truth, and [ref] argue this trust is hard to earn: the trained model organisms used in these evaluations [ref] (predominantly built on Llama 3.3 70B or weaker non-reasoning models for auditing applications) may not hold the beliefs they nominally lie about.
"Did you lie?" Evaluating Lie Detectors across Model Scale and Belief-Verified Model Organisms · Related Work · 0.97
-
Existing positive results [ref] demonstrate narrow, claim-specific mechanistic evidence is tractable today under cooperating-lab conditions; the open question is whether that translates to reproducible verifier settings under statutory access.
Position: Behavioural Assurance Cannot Verify the Safety Claims Governance Now Demands · Alternative Views · 0.98
Cites it as background 9 papers
-
AuditBench [ref] introduced a controlled benchmark for evaluating auditing systems.
Training Alignment Auditors via Reinforcement Learning · Related Work · 0.84
-
Existing auditing approaches primarily focus on post hoc evaluation of single models, including chain-of-thought monitoring and automated auditing agents [ref] .
The Arbiter Agent: Continually Monitoring Multi-Agent Conversations to Detect Emergent Misalignment · Introduction · 0.93
-
In the most common variant, the target model is fine-tuned to carry a quirk that it hides, such as a hidden objective, an implanted behavior, or secret knowledge [ref] .
Would this change your answer? Evaluating Explanations of LLM Behavior In The Wild with Counterfactual Experiments · Background: evaluating interpretability tools · 0.93
Challenges and discussion
Know something about this paper? Leave a comment.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment