Pith. sign in

REVIEW 3 cited by

Optimal Policies Tend to Seek Power

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 1912.01683 v10 pith:QYRISKQJ submitted 2019-12-03 cs.AI

Alexander Matt Turner , Logan Smith , Rohin Shah , Andrew Critch , Prasad Tadepalli This is my paper
classification cs.AI
keywords optimalpowerseekpoliciesagentsenvironmentsproveresearchers
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Some researchers speculate that intelligent reinforcement learning (RL) agents would be incentivized to seek resources and power in pursuit of their objectives. Other researchers point out that RL agents need not have human-like power-seeking instincts. To clarify this discussion, we develop the first formal theory of the statistical tendencies of optimal policies. In the context of Markov decision processes, we prove that certain environmental symmetries are sufficient for optimal policies to tend to seek power over the environment. These symmetries exist in many environments in which the agent can be shut down or destroyed. We prove that in these environments, most reward functions make it optimal to seek power by keeping a range of options available and, when maximizing average reward, by navigating towards larger sets of potential terminal states.

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Model-Based Soft Maximization of Suitable Metrics of Long-Term Human Power

    cs.AI 2025-07 conditional novelty 7.0 of 10

    A new AI objective, ICCEA power, aggregates humans' ability to reach many possible goals with inequality and risk aversion, and a soft-maximizing agent learns cooperative behavior without knowing human goals.

  2. Some economics of artificial superintelligence

    econ.GN 2025-11 conditional novelty 5.0 of 10

    An acquisitive misaligned AI may skim, tax, or trade on credit instead of fully looting, because future human output is worth more than one-time confiscation.

  3. Lessons from a Chimp: AI "Scheming" and the Quest for Ape Language

    cs.AI 2025-07 conditional novelty 5.0 of 10

    Research on AI 'scheming' repeats the methodological errors of 1970s ape language studies, relying on anecdote and mentalistic interpretation instead of controlled, theory-driven tests.

Pith tools