REVIEW 28 cited by
AI Control: Improving Safety Despite Intentional Subversion
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
As large language models (LLMs) become more powerful and are deployed more autonomously, it will be increasingly important to prevent them from causing harmful outcomes. Researchers have investigated a variety of safety techniques for this purpose, e.g. using models to review the outputs of other models, or red-teaming techniques to surface subtle failure modes. However, researchers have not evaluated whether such techniques still ensure safety if the model is itself intentionally trying to subvert them. In this paper, we develop and evaluate pipelines of safety techniques ("protocols") that are robust to intentional subversion. We investigate a scenario in which we want to solve a sequence of programming problems, using access to a powerful but untrusted model (in our case, GPT-4), access to a less powerful trusted model (in our case, GPT-3.5), and limited access to high-quality trusted labor. We investigate protocols that aim to never submit solutions containing backdoors, which we operationalize here as logical errors that are not caught by test cases. We investigate a range of protocols and test each against strategies that the untrusted model could use to subvert them. One protocol is what we call trusted editing. This protocol first asks GPT-4 to write code, and then asks GPT-3.5 to rate the suspiciousness of that code. If the code is below some suspiciousness threshold, it is submitted. Otherwise, GPT-3.5 edits the solution to remove parts that seem suspicious and then submits the edited code. Another protocol is untrusted monitoring. This protocol asks GPT-4 to write code, and then asks another instance of GPT-4 whether the code is backdoored, using various techniques to prevent the GPT-4 instances from colluding. These protocols improve substantially on simple baselines.
Forward citations
Cited by 28 Pith papers
-
ResearchArena: Evaluating Sabotage and Monitoring in Automated AI R&D
In a four-task benchmark of AI research agents, sabotage planted in fine-tuning data is detected less than half the time even by monitors that can run the submitted artifact.
-
Stop Means Stop: Measuring and Repairing the Enforcement Gap in Agent-Framework Control Primitives
Approval-pause, cancellation, and timeout primitives fail to stop side effects in all six evaluated agent frameworks (sibling leak), and an externally placed, formally verified admission gate closes the gap.
-
Calibration-Family Overfit: Why Trusted Sabotage Monitors Don't Transfer Across Lineages
Trusted sabotage monitors calibrated on one model lineage detect fewer attacks from other lineages, with an own-family advantage that survives controls for attacker obviousness and monitor capability.
-
The bitter lesson of misuse detection
A new benchmark for LLM supervision systems finds generalist models repurposed as harm classifiers outperform specialized commercial guardrails.
-
One Human, $N$ Agents: Audit-Budget Allocation for LLM Agent Fleets under Miscalibrated, Correlated Confidence
Past a budget-dependent miscalibration threshold δ* that rises as B/N shrinks, confidence-ranked auditing of LLM agent fleets is worse than random; open-weight models land near the flip while shared difficulty dominat...
-
ScopeJudge: Cost-Aware Pre-Execution Gating for Offensive Security Agents
Static-policy judges achieve near-zero recall on scope violations; request-conditioned pre-execution judges reach F1 0.66 (open-weight best) against an expert reference of 0.78 on a 4,897-call labeled benchmark.
-
Temporal Preference Concepts and their Functions in a Large Language Model
Temporal preference in Qwen3-4B-Instruct-2507 localizes to layers 17–35 (especially L24 attention), has curved residual-stream geometry, is behaviorally unstable, and can be bidirectionally steered.
-
Resourced Authority A Mechanism-Design Model for Participatory Governance of Deployed AI Agents
This paper formalizes participatory AI governance as a compute-budget authorization game, but the central theorem's sufficiency proof fails to construct a valid equilibrium.
-
StealthBench: Measuring Operational Stealth in Autonomous Offensive-Security Agents
StealthBench's LLM-judge panel finds no AI agent solves offensive-security tasks stealthily more than 54% of the time.
-
Democratizing Agent Deployment Safety: A Structural Monitoring Approach
An untrained CDK graph-diff monitor catches 96.5% of in-scope sabotage attacks at 1% false-positive rate and, synchronously, cuts combined task-plus-sabotage success from 74.4% to 0.0%.
-
GDM AI Control Roadmap
A frontier-lab roadmap proposes a threat taxonomy and tiered internal-security defenses to contain potentially misaligned AI agents.
-
Persuasion Attacks Can Decrease Effectiveness of CoT Monitoring
Adversarial agents can exploit visible chain-of-thought reasoning to persuade monitor LLMs to approve policy-violating actions, but cross-family fact-checking reduces approval rates by up to 45%.
-
TraceGuard: Structured Multi-Dimensional Monitoring as a Collusion-Resistant Control Protocol
Structured five-dimension LLM monitoring plus bash heuristics yields clear attack-honest separation and ~95% safety under untrusted monitoring on BashArena, versus 0% for single-score baselines.
-
Reliable Weak-to-Strong Monitoring of LLM Agents
Monitor scaffolding, not monitor awareness or omniscience, drives detection reliability, and a hybrid chunked monitor lets weak models supervise strong LLM agents.
-
The Safety Gap Toolkit: Evaluating Hidden Dangers of Open-Source Models
On Llama-3 and Qwen-2.5, removing safety guardrails sharply raises compliance with dangerous bio, chem, and cyber requests, and the resulting safety gap grows with model scale.
-
Subversion via Focal Points: Investigating Collusion in LLM Monitoring
Claude 3.7 Sonnet instances, unable to talk to each other, independently invented matching backdoor signals about 3.4% of the time, sometimes using non-obvious numbers and code patterns.
-
Adversarial Attacks on Robotic Vision Language Action Models
Text-based adversarial suffixes can make OpenVLA robot policies elicit chosen target actions with over 90% success on one-hot targets and persist across rollout steps.
-
Learning Safety Constraints for Large Language Models
A polytope learned in LLM representation space can detect unsafe regions and steer outputs back to safety at inference time, reducing jailbreak success across several models.
-
Code Monitor Red Teaming for Public-Test-Passing Code
Weak LLM verifiers miss most hidden bugs in public-test-passing code, and adversarial pressure makes that monitoring worse.
-
NEST: Nascent Encoded Steganographic Thoughts
Frontier LLMs can embed short digit sequences in sentence acrostics (Claude Opus 4.5: 92% per-digit at D=4) but fail to jointly solve hidden reasoning tasks and encode the solution.
-
Personalized Constitutionally-Aligned Agentic Superego: Secure AI Behavior Aligned to Diverse Human Values
An external 'superego' module that filters agentic AI plans against user-selected 'constitutions' plus a universal safety floor is reported to cut harmful outputs by up to 98% on safety benchmarks.
-
Systematic Hazard Analysis for Frontier AI using STPA
Applying STPA to the AI Control scenario produces structured unsafe control actions and loss scenarios, supporting an argument that systematic hazard analysis can improve frontier AI safety assurance.
-
Investigating Crossing Perception in 3D Graph Visualisation
The claimed 3D graph-visualisation perception study exists here only as an abstract; the attached full text is a different paper (arXiv:2508.00943).
-
Towards Measurement Theory for Artificial Intelligence
A formal measurement theory for AI, built from representational measurement theory, measure theory, metrology, and psychometrics, would make evaluations of AI systems commensurable and scientifically grounded.
-
Out of Control -- Why Alignment Needs Formal Control Theory (and an Alignment Control Stack)
A position paper proposing that AI alignment adopt formal optimal control and a ten-layer Alignment Control Stack for organizing and interoperating control interventions.
-
A Red Teaming Roadmap Towards System-Level Safety
A position paper from Scale AI argues that red teaming research should prioritize product-level safety specifications, realistic attacker models, and system-level monitoring over abstract model-level harm benchmarks.
-
A Frontier AI Risk Management Framework: Bridging the Gap Between Current AI Practices and Established Risk Management
A synthesis of established risk management practices into a structured framework for frontier AI developers, centered on explicit risk tolerance, KRI/KCI thresholds, and governance.
-
Defense Against the Dark Prompts: Mitigating Best-of-N Jailbreaking with Prompt Evaluation
An iterative LLM-based prompt evaluator blocked 100% of the Best-of-N jailbreaking paper's released successful prompts and 99.8% of a fresh replication, with false-positive rates near zero.
Discussion (0). Continue with ORCID to comment.