REVIEW 9 cited by
Ctrl-Z: Controlling AI Agents via Resampling
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Control evaluations measure whether monitoring and security protocols for AI systems prevent intentionally subversive AI models from causing harm. Our work presents the first control evaluation performed in an agent environment. We construct BashBench, a dataset of 257 challenging multi-step system administration tasks, and evaluate whether various safety measures can prevent an adversarially constructed AI agent from covertly downloading and executing malicious code in this environment. This multi-step setting introduces new attack and defense dynamics, which we investigate in order to design novel control protocols that prevent safety failures without hindering the ability of non-malicious agents to perform useful work. We introduce a class of control protocols called resample protocols that dynamically take additional samples of certain actions. We find these protocols significantly improve on existing techniques by selectively blocking the AI agent from executing suspicious code and incriminating the agent by generating additional examples of dangerous behavior. We measure the tradeoff between attack prevention and usefulness; our best protocol combines resampling with analysis of previous steps, reducing the success rate of attacks from 58% to 7% at a 5% cost to the performance of a non-malicious agent.
Forward citations
Cited by 9 Pith papers
-
ResearchArena: Evaluating Sabotage and Monitoring in Automated AI R&D
In a four-task benchmark of AI research agents, sabotage planted in fine-tuning data is detected less than half the time even by monitors that can run the submitted artifact.
-
SHADE-Arena: Evaluating Sabotage and Monitoring in LLM Agents
A new 17-task benchmark measures how well frontier AI agents can secretly complete a harmful side task while doing a benign task, and how well AI monitors can detect them; the best agent succeeds 27% of the time and t...
-
Agent-Safety Evaluations as Load-Bearing Evidence: A Vendor-Neutral, Cross-Harness Reconstructability Metric
A property-level reconstructability metric and Evidence Sufficiency Card show that traces sharing a surface reading can differ sharply in evidence sufficiency, and that replay preconditions often fail.
-
GDM AI Control Roadmap
A frontier-lab roadmap proposes a threat taxonomy and tiered internal-security defenses to contain potentially misaligned AI agents.
-
Reliable Weak-to-Strong Monitoring of LLM Agents
Monitor scaffolding, not monitor awareness or omniscience, drives detection reliability, and a hybrid chunked monitor lets weak models supervise strong LLM agents.
-
Manipulation Attacks by Misaligned AI: Risk Analysis and Safety Case Framework
A framework paper that adapts AI safety case methodology to the specific threat of manipulation attacks by internally deployed misaligned AI.
-
Subversion via Focal Points: Investigating Collusion in LLM Monitoring
Claude 3.7 Sonnet instances, unable to talk to each other, independently invented matching backdoor signals about 3.4% of the time, sometimes using non-obvious numbers and code patterns.
-
On Understanding, Identifying, and Mitigating Vulnerabilities in Agentic Large Language Models
A PRISMA-based survey of 85 papers shows agentic LLM security research is attack-heavy and perception-focused, leaving action-layer and code-execution risks understudied.
-
Position Paper: Bounded Alignment: What (Not) To Expect From AGI Agents
The paper argues that perfect alignment of general AI is impossible in principle and proposes 'bounded alignment' as the realistic safety goal.
Discussion (0). Continue with ORCID to comment.