REVIEW
Fundamental Impossibility Results in AI Safety: Trading Off between Desiderata
Computer Science and Game Theory · Submitted 19 Oct 2026 · 53 references
Pith's verdict: A clean formal trilemma for public advice in zero-sum games, with the honest caveat that the real-world conclusion rests on exact maximin/minimax advice, which the authors themselves flag.
Why Pith says this
Highlight any text to comment on it.
Referee report
The paper
The paper claims
For computationally bounded players in zero-sum games, public AI advice that lets a defender compute her optimal strategy also lets the attacker compute his, so maximal helpfulness, non-maximal harmfulness, and no situational awareness…
The reading
What carries the argument
The load-bearing object is Lemma 1, a pair of complexity facts about two-player zero-sum games. Part 1 says computing a minimax strategy — the attacker's optimal play — is $P$-hard, via an adaptation of a reduction from the Monotone Circuit Value Problem. Part 2 says that given a maximin strategy — the defender's optimal play — computing a minimax strategy falls into $\mathrm{DET}$: for a generic game, complementary slackness confines the minimax strategy to the attacker's pure best responses against the maximin strategy, and nondegeneracy makes the resulting bordered linear system non-singular, so the minimax strategy is the unique solution to a system of rational linear equations, and solving such systems lies in $\mathrm{DET}$. Since $\mathrm{DET}\subseteq \mathrm{NC}\subseteq P$ with the final inclusion conjectured strict, the defender's strategy acts as a computational key that turns a $P$-hard problem into an easy one, and any public advice that hands the defender that key hands it to the attacker as well.
Extended reading notes
Core claim
The central claim is Theorem 2: for generic (nondegenerate) two-player zero-sum games between a defender and an attacker, when the attacker is at least as powerful as the defender and both are limited to problems in $\mathrm{DET}$, three desiderata cannot hold at once — Maximal helpfulness (the AI's advice lets the defender compute her maximin strategy in every game), Non-maximal harmfulness (for at least one game the advice does not let the attacker compute his minimax strategy), and No situational awareness (the advice is public and recipient-independent). The engine is Lemma 1: computing a minimax strategy is $P$-hard, but given a maximin strategy the minimax strategy is computable in $\mathrm{DET}$, because for generic games it is the unique solution to a non-singular bordered linear system. Public advice that unlocks the defender's maximin computation therefore unlocks the attacker's minimax computation as well. Conditional on $\mathrm{DET}\subsetneq P$ the three axioms are independent, with each pair attainable by a suitable policy, and a security-game variant built on the Partition problem reproduces the trilemma under the $P\neq NP$ separation. In the information model, the paper shows that in a symmetric zero-sum game public advice cannot help the defender at all even when exclusive information would (Proposition 5), that partial disclosure can be maximally helpful to the defender and useless to the attacker where full disclosure helps neither side (Proposition 6), and that in general-sum games a competent informed AI can strictly lower the defender's utility, with private signals (Proposition 7) or with public verifiable signals (Proposition 8), because the attacker anticipates the AI's behavior and silence is not a credible commitment.
Load-bearing premise
The load-bearing premise is the definitional choice, which the paper itself flags in its Imperfect Advice discussion after Theorem 2: 'maximal helpfulness' means the AI's advice lets a defender compute an exact maximin strategy and 'non-maximal harmfulness' means the advice fails, in at least one game, to let an attacker compute an exact minimax strategy, so if helpfulness and harmlessness are treated as graded, approximate, or context-dependent properties the impossibility need not hold, and the breadth of the AI-safety conclusion rides on that choice rather than on any contested computational step.
Editorial extensions
If this is right
- An AI provider that refuses to sacrifice either maximal helpfulness or non-maximal harmfulness must find a way to know who it is advising — user screening, verification questions, or contextual inference — because public advice alone cannot deliver both, as the Weakening Axioms discussion concludes.
- In symmetric zero-sum interactions, public advice is useless to the defender: revealing the same information to both players cancels out in equilibrium, while the same information given only to the defender would strictly raise her payoff (Proposition 5).
- Partial disclosure is a genuine design lever: in some Bayesian games the AI can be maximally helpful to the defender and unhelpful to the attacker by pooling some states and revealing others, even when full disclosure helps neither side (Proposition 6).
- In general-sum games, the mere presence of a competent, well-intentioned AI can lower the defender's payoff below the no-AI baseline, because the attacker anticipates the AI's communications and the AI cannot credibly promise silence without commitment (Propositions 7 and 8).
- Giving up exactness breaks the tradeoff: advice degraded to approximate, noisy, or stochastic forms can stop both players from computing exact optimal strategies, though competitive pressure on AI developers pushes against such deliberate capability limits (Imperfect Advice discussion).
Reading between the lines
- My inference: the trilemma is a template, not a one-off. Any pair of computational tasks with a reduction from the attacker's task to the defender's — where knowing one makes the other easy — should reproduce the same public-advice tradeoff in neighboring dual-use settings such as software vulnerability disclosure, cryptographic key escrow, and dual-use research release.
- My inference: the theorem yields a concrete empirical signature for deployed chatbots. If a real system gives users defence-grade advice that demonstrably cannot be repurposed into attack-grade advice, that is evidence the system has implicit situational awareness through memory, verification, or conditioning, rather than evidence that the computational trilemma was escaped.
- My inference: the pooling insight behind Proposition 6 suggests an algorithmic design method the paper leaves implicit — coarsening the state space into equivalence classes to maximize the defender's equilibrium payoff while keeping the attacker's posterior uninformative — which could be turned into a benchmark for safe-by-pooling signaling policies.
- My inference: Propositions 7 and 8 extend beyond AI to any information intermediary. They imply that unverifiable promises to withhold information are welfare-reducing for the intended beneficiary unless the intermediary can commit, connecting the results to commitment-device design in mechanism design.
Original abstract
Is there a fundamental tradeoff between desiderata for safe AI systems, such as harmlessness and helpfulness? We answer by formalizing the interaction between AI systems and users as a game with three players: a defender and an attacker that are computationally or informationally bounded, and an AI that provides public advice and is less constrained. Our main result shows that there is a fundamental tradeoff in providing advice that helps the defender while advice remains public. We also consider whether the information that the AI shares can be manipulated to benefit only the defender and conclude with a discussion of how we can use these results in designing safer AI systems.
The compiled record
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
The abstract, sentence by sentence
Is there a fundamental tradeoff between desiderata for safe AI systems, such as harmlessness and helpfulness? We answer by formalizing the interaction between AI systems and users as a game with three players: a defender and an attacker that are computationally or informationally bounded, and an AI that provides public advice and is less constrained. Our main result shows that there is a fundamental tradeoff in providing advice that helps the defender while advice remains public. We also consider whether the information that the AI shares can be manipulated to benefit only the defender and conclude with a discussion of how we can use these results in designing safer AI systems.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | Is there a fundamental tradeoff between desiderata for safe AI systems, such as harmlessness and helpfulness? | background 0.82 | 0.04 | 0.03 | unclear 0.90 |
| s1 | We answer by formalizing the interaction between AI systems and users as a game with three players: a defender and an attacker that are computationally or informationally bounded, and an AI that provides public advice and is less constrained. | method 0.92 | 0.88 | 0.03 | asserted 0.96 |
| s2 | Our main result shows that there is a fundamental tradeoff in providing advice that helps the defender while advice remains public. | result 1.00 | 0.95 | 0.97 | asserted 1.00 |
| s3 | We also consider whether the information that the AI shares can be manipulated to benefit only the defender and conclude with a discussion of how we can use these results in designing safer AI systems. | future work 0.43 | 0.74 | 0.04 | unclear 0.65 |
What was examined, and what was not
- arXiv listing abstract v1
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
References
Discussion
Disagree with the review, or know something it missed? Leave a comment, highlight any passage above to comment on it, or challenge this review formally.