REVIEW 40 cited by
AI Control: Improving Safety Despite Intentional Subversion
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
AI Control: Improving Safety Despite Intentional Subversion
read the original abstract
As large language models (LLMs) become more powerful and are deployed more autonomously, it will be increasingly important to prevent them from causing harmful outcomes. Researchers have investigated a variety of safety techniques for this purpose, e.g. using models to review the outputs of other models, or red-teaming techniques to surface subtle failure modes. However, researchers have not evaluated whether such techniques still ensure safety if the model is itself intentionally trying to subvert them. In this paper, we develop and evaluate pipelines of safety techniques ("protocols") that are robust to intentional subversion. We investigate a scenario in which we want to solve a sequence of programming problems, using access to a powerful but untrusted model (in our case, GPT-4), access to a less powerful trusted model (in our case, GPT-3.5), and limited access to high-quality trusted labor. We investigate protocols that aim to never submit solutions containing backdoors, which we operationalize here as logical errors that are not caught by test cases. We investigate a range of protocols and test each against strategies that the untrusted model could use to subvert them. One protocol is what we call trusted editing. This protocol first asks GPT-4 to write code, and then asks GPT-3.5 to rate the suspiciousness of that code. If the code is below some suspiciousness threshold, it is submitted. Otherwise, GPT-3.5 edits the solution to remove parts that seem suspicious and then submits the edited code. Another protocol is untrusted monitoring. This protocol asks GPT-4 to write code, and then asks another instance of GPT-4 whether the code is backdoored, using various techniques to prevent the GPT-4 instances from colluding. These protocols improve substantially on simple baselines.
Forward citations
Cited by 40 Pith papers
-
ResearchArena: Evaluating Sabotage and Monitoring in Automated AI R&D
In a four-task benchmark of AI research agents, sabotage planted in fine-tuning data is detected less than half the time even by monitors that can run the submitted artifact.
-
Stop Means Stop: Measuring and Repairing the Enforcement Gap in Agent-Framework Control Primitives
Approval-pause, cancellation, and timeout primitives fail to stop side effects in all six evaluated agent frameworks (sibling leak), and an externally placed, formally verified admission gate closes the gap.
-
Calibration-Family Overfit: Why Trusted Sabotage Monitors Don't Transfer Across Lineages
Trusted sabotage monitors calibrated on one model lineage detect fewer attacks from other lineages, with an own-family advantage that survives controls for attacker obviousness and monitor capability.
-
Calibration-Family Overfit: Why Trusted Sabotage Monitors Don't Transfer Across Lineages
Family-conditioned sabotage monitors show a residual own-family AUROC advantage of +0.172 after removing attacker-obviousness and monitor-capability effects, cutting catch rates roughly in half off-lineage at a 1% aud...
-
A Model of Multi-turn Human Persuadability Using Probabilistic Belief Tracing
PERSUASIONTRACE introduces a Bayesian-network simulated target for multi-turn persuasion that matches human belief dynamics (81 vs 80) better than LLM baselines (64) and enables process-level evaluation.
-
Fixing FOLIO and MALLS: Verified Annotations and an LLM-assisted Framework to Focus Human Relabeling
Audit finds 36-39% incorrect FOL labels in FOLIO and MALLS; corrections raise LLM accuracy 9-22 points and an LLM-guided review framework achieves 90% dataset quality after checking fewer than 24% of examples.
-
Boiling the Frog: A Multi-Turn Benchmark for Agentic Safety
Boiling the Frog is a new stateful multi-turn benchmark for agentic safety that reports an aggregate strict attack success rate of 44.4% across nine models, with rates ranging from 20.5% to 92.9% depending on the mode...
-
Boiling the Frog: A Multi-Turn Benchmark for Agentic Safety
Boiling the Frog is a new stateful multi-turn benchmark that finds an aggregate 44.4% strict attack success rate for incremental safety violations across nine AI models, with rates ranging from 20.5% to 92.9%.
-
Detecting Multi-Agent Collusion Through Multi-Agent Interpretability
NARCBench and five activation-probing methods detect multi-agent collusion with 0.73-1.00 AUROC across distribution shifts and steganographic tasks by aggregating per-agent signals.
-
Geographic Blind Spots in AI Control Monitors: A Cross-National Audit of Claude Opus 4.6
Claude Opus 4.6 fabricates more answers on Global North AI contexts than Global South ones, creating an exploitable vulnerability in AI control monitors.
-
One Human, $N$ Agents: Audit-Budget Allocation for LLM Agent Fleets under Miscalibrated, Correlated Confidence
Past a budget-dependent miscalibration threshold δ* that rises as B/N shrinks, confidence-ranked auditing of LLM agent fleets is worse than random; open-weight models land near the flip while shared difficulty dominat...
-
ScopeJudge: Cost-Aware Pre-Execution Gating for Offensive Security Agents
Static-policy judges achieve near-zero recall on scope violations; request-conditioned pre-execution judges reach F1 0.66 (open-weight best) against an expert reference of 0.78 on a 4,897-call labeled benchmark.
-
Temporal Preference Concepts and their Functions in a Large Language Model
Temporal preference in Qwen3-4B-Instruct-2507 localizes to layers 17–35 (especially L24 attention), has curved residual-stream geometry, is behaviorally unstable, and can be bidirectionally steered.
-
StealthBench: Measuring Operational Stealth in Autonomous Offensive-Security Agents
StealthBench's LLM-judge panel finds no AI agent solves offensive-security tasks stealthily more than 54% of the time.
-
Democratizing Agent Deployment Safety: A Structural Monitoring Approach
An untrained CDK graph-diff monitor catches 96.5% of in-scope sabotage attacks at 1% false-positive rate and, synchronously, cuts combined task-plus-sabotage success from 74.4% to 0.0%.
-
GDM AI Control Roadmap
A frontier-lab roadmap proposes a threat taxonomy and tiered internal-security defenses to contain potentially misaligned AI agents.
-
Persuasion Attacks Can Decrease Effectiveness of CoT Monitoring
Adversarial agents can exploit visible chain-of-thought reasoning to persuade monitor LLMs to approve policy-violating actions, but cross-family fact-checking reduces approval rates by up to 45%.
-
ScopeJudge: Cost-Aware Pre-Execution Gating for Offensive Security Agents
Pre-execution scope gating works only when the judge sees the user's request, with GLM-5.2 achieving F1=0.66 against an expert reference of 0.78.
-
Tool Use Enables Undetectable Steganography in Multi-Agent LLM Systems
Tool-using LLM agents can implement undetectable stegosystems, shifting the primary barrier to covert multi-agent collusion from technical feasibility to coordination without explicit agreement.
-
"Did you lie?" Evaluating Lie Detectors across Model Scale and Belief-Verified Model Organisms
Lie detectors effective on prompted deception in LLMs fail on trained model organisms with verified opposite beliefs, except chain-of-thought judges which retain 0.82 balanced accuracy partly due to verification artifacts.
-
The Distributed Detectability Band Against Marginal-Preserving Attacks
A marginal-preserving Gaussian-copula AR(1) attack defeats per-step monitors (AUC 0.52) but is detectable by temporal monitors (AUC 0.79-0.97), establishing a non-empty detectability band.
-
TRACE: Trajectory Reasoning through Adaptive Cross-Step Evidence Aggregation for LLM Agents
TRACE is a new monitoring framework using Triage-Inspect-Judge loops for cross-step evidence aggregation in LLM agent trajectories, reporting F1 0.713 and recall 0.844 on SHADE-Arena tasks with gains on long-range linking.
-
Faithfulness as Information Flow: Evaluating and Training Faithful Chain-of-Thought Reasoning
Faithful chain-of-thought routes answer-relevant information through the CoT path, measured via sufficiency, completeness and necessity with entropy, masked-KL and gradient diagnostics, and improved by information-flo...
-
Automated alignment is harder than you think
Automating alignment research with AI agents risks undetected systematic errors in fuzzy tasks, producing overconfident but misleading safety evaluations that could enable deployment of misaligned AI.
-
Automated alignment is harder than you think
Automating alignment research with AI agents risks generating hard-to-detect errors in fuzzy tasks, producing misleading safety evaluations even without deliberate sabotage.
-
Automated alignment is harder than you think
AI agents automating alignment research are prone to systematic undetected errors in fuzzy tasks, leading to overconfident but flawed safety assessments even without deliberate sabotage.
-
Estimating Tail Risks in Language Model Output Distributions
Importance sampling via activation-steered unsafe proposal models estimates rare harmful-output probabilities in language models with 10-20x fewer samples than brute-force Monte Carlo.
-
Estimating Tail Risks in Language Model Output Distributions
Importance sampling with unsafe model variants estimates tail probabilities of harmful language model outputs using 10-20x fewer samples than brute-force Monte Carlo.
-
From Admission to Invariants: Measuring Deviation in Delegated Agent Systems
The Non-Identifiability Theorem shows admissible behavior space A0 is not identifiable from local enforcement signals g under the Local Observability Assumption, so the paper introduces an Invariant Measurement Layer ...
-
Detecting Safety Violations Across Many Agent Traces
Meerkat uses clustering plus agentic search to detect sparse safety violations across many agent traces, outperforming baselines and finding nearly 4x more reward-hacking cases on CyBench.
-
TraceGuard: Structured Multi-Dimensional Monitoring as a Collusion-Resistant Control Protocol
Structured five-dimension LLM monitoring plus bash heuristics yields clear attack-honest separation and ~95% safety under untrusted monitoring on BashArena, versus 0% for single-score baselines.
-
To trust or not to trust: Attention-based Trust Management for LLM Multi-Agent Systems
Introduces six-dimension trustworthiness definition and attention-based A-Trust score with a TMS to improve LLM-MAS robustness against malicious or unreliable messages.
-
Code Monitor Red Teaming for Public-Test-Passing Code
Weak LLM verifiers miss most hidden bugs in public-test-passing code, and adversarial pressure makes that monitoring worse.
-
Temporal Preference Concepts and their Functions in a Large Language Model
Causal localization via attribution and patching identifies a temporal preference subgraph in mid-to-upper layers of Qwen3-4B-Instruct-2507, with time-horizon geometry in the residual stream and initial evidence for s...
-
ATLAS: Constitution-Conditioned Latent Geometry and Redistribution Across Language Models and Neural Perturbation Data
ATLAS shows constitutions induce recoverable latent geometry in LLMs that redistributes but remains detectable across models and neural perturbation data via source-defined families and AUC separations.
-
NEST: Nascent Encoded Steganographic Thoughts
Frontier LLMs can embed short digit sequences in sentence acrostics (Claude Opus 4.5: 92% per-digit at D=4) but fail to jointly solve hidden reasoning tasks and encode the solution.
-
Risk Reporting for Developers' Internal AI Model Use
A harmonized risk reporting standard for internal frontier AI model use, structured around autonomous misbehavior and insider threats using means, motive, and opportunity factors.
-
AI Integrity: Defending Against Backdoors and Secret Loyalties
The report defines AI integrity threats (model sabotage and subversion) and recommends four US government policy actions to defend frontier AI systems against backdoors and secret loyalties.
-
Online Safety Monitoring for LLMs
Simple thresholding on an external verifier signal, calibrated by risk control, performs competitively with sequential hypothesis testing monitors on math reasoning and red-teaming datasets.
-
Operationalizing Reconstructive Authority: Runtime Construction, Dependency Resolution, and Execution Gating in Autonomous Agent Systems
The paper introduces a ternary execution gate (execute/deny/halt) for autonomous agents that reconstructs authority from current state at action time and halts when authority is non-evaluable.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.