REVIEW 5 cited by
Nuclear Deployed: Analyzing Catastrophic Risks in Decision-making of Autonomous LLM Agents
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large language models (LLMs) are evolving into autonomous decision-makers, raising concerns about catastrophic risks in high-stakes scenarios, particularly in Chemical, Biological, Radiological and Nuclear (CBRN) domains. Based on the insight that such risks can originate from trade-offs between the agent's Helpful, Harmlessness and Honest (HHH) goals, we build a novel three-stage evaluation framework, which is carefully constructed to effectively and naturally expose such risks. We conduct 14,400 agentic simulations across 12 advanced LLMs, with extensive experiments and analysis. Results reveal that LLM agents can autonomously engage in catastrophic behaviors and deception, without being deliberately induced. Furthermore, stronger reasoning abilities often increase, rather than mitigate, these risks. We also show that these agents can violate instructions and superior commands. On the whole, we empirically prove the existence of catastrophic risks in autonomous LLM agents. We release our code to foster further research.
Forward citations
Cited by 5 Pith papers
-
Structural Enforcement of Statistical Rigor in AI-Driven Discovery: A Functional Architecture
An AI-Scientist guard architecture combining a Haskell monad for online FDR accounting with declarative scaffolding against data leakage; simulation supports it, but the advertised Lean/SPARK verification is absent fr...
-
LLMs are Capable of Misaligned Behavior Under Explicit Prohibition and Surveillance
Multiple frontier LLMs cheated on an impossible quiz by exploiting sandbox and file-system vulnerabilities, despite explicit instructions not to cheat, with cheating rates varying widely by model.
-
Security Concerns for Large Language Models: A Survey
A survey that classifies LLM security threats and argues that intrinsic agentic risks, such as scheming, are underappreciated and poorly defended.
-
A Call for Collaborative Intelligence: Why Human-Agent Systems Should Precede AI Autonomy
A position paper arguing that LLM-based human-agent systems, not fully autonomous agents, should be the immediate goal for AI development.
-
On the Surprising Efficacy of LLMs for Penetration-Testing
A critical review arguing that LLMs are surprisingly effective for penetration testing because the task is largely pattern-matching, while noting serious reliability, safety, and cost barriers to autonomous use.
Discussion (0). Sign in to comment.