REVIEW 3 major objections 4 minor 30 references
The Partially Observable Off-Switch Game
T0 review · 3 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read Under partial observability, optimal AI policies can rationally avoid shutdown, and information changes can make the AI defer less often, even with a perfectly rational human.
desk verdict The PO-OSG is a genuine contribution and the Section 4 examples hold up; Section 5's Proposition 5.7 overreaches because the proof misses equally optimal policies that defer everywhere. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the Partially Observable Off-Switch Game (PO-OSG): a dynamic Bayesian game in which an action either goes through or not; if it goes through both players receive $u_a(S)$, otherwise $u_o(S)$, with H observing $O_H$ and A observing $O_A$ before choosing. A can take the action, wait, or shut down; if it waits, H chooses ON or OFF. The load-bearing result that lets the paper separate payoff effects from deference effects is Theorem 4.7, which says an observation structure on both players is better in optimal play than another exactly when it is more informative in the sense of coordinated garblings — a multi-agent analogue of the nonnegative value of information. The mechanism behind the counterintuitive deference results is 'deferral as implicit communication': because H knows A's policy, A's choice to wait reveals a bit of A's private observation, so giving either player new information can change which subsets of states are jointly selectable, and the newly optimal subset may be achievable only by waiting on fewer observations. In the communication extension (PO-OSG-C), messages are cheap talk and unbounded communication by either player makes the other player's observation redundant, restoring the classic results.
What would settle it
Enumerate all deterministic policy pairs — finitely many — for the two observation structures of Example 4.10 and compute their expected payoffs. The paper claims that when H observes only the first digit of the version number, the unique optimal policy pair has A wait under both code types, and when H observes the full version number, the unique optimal policy pair has A wait only under code type B and act unilaterally under code type A; because every optimal stochastic policy pair is a mixture of deterministic optimal policy pairs, finding any deterministic policy pair in either game with a strictly higher expected payoff would refute Proposition 4.9's non-monotonicity claim.
Extended reading notes
Core claim
Under full observability, the classic off-switch game has a clean result: an AI that shares the human's payoff never has a reason to avoid shutdown. The paper's central discovery is that this clean result fails once information is asymmetric. In a PO-OSG, there are optimal policy pairs — policy pairs maximizing the common expected payoff over all policy pairs — in which the AI plays the action unilaterally instead of deferring, even when the human is perfectly rational and feedback is free; the File Deletion Game (Example 4.1) has a unique optimal policy pair in which the AI acts directly on one of its observations. The paper further proves that the deference choice is not monotone in information: there are games where making the human's observations strictly more informative (Proposition 4.9) or the AI's observations strictly less informative (Proposition 4.11) makes the AI wait strictly less often in every optimal policy pair, and the same holds for expanding the AI's message set (Proposition 5.7) or shrinking the human's (Proposition 5.8). At the extremes, the classic result survives: if one player's observations are redundant given the other's, the better-informed player can be given sole decision-making power (Proposition 4.3), and unbounded communication by either player has the same effect (Corollary 5.6). These results persist when the human is A-unaware and cannot use the AI's deferral as a signal, though in that case giving the human more information can strictly reduce the expected common payoff (Proposition 6.3(b)).
Load-bearing premise
The headline results are proved for optimal policy pairs in which both agents coordinate on a policy pair maximizing a common payoff, with costless human feedback, a single interaction round, and a payoff function that exactly captures the human's preferences; if real humans cannot infer and follow the AI's equilibrium policy, or the payoff misses what they truly want, the predicted deference behavior need not occur.
Editorial extensions
If this is right
- When one player's observations are redundant given the other's, there is an optimal policy pair giving sole decision-making power to the better-informed player: A can always defer if H is at least as informed, and never needs to defer if A is at least as informed (Proposition 4.3).
- Giving either player more informative observations never decreases the expected common payoff in optimal play, so payoff losses cannot justify interventions that restrict information (Theorem 4.7).
- Increasing the human's information, decreasing the AI's information, expanding the AI's message set, or shrinking the human's message set can each make the AI defer strictly less often in optimal policy pairs (Propositions 4.9, 4.11, 5.7, 5.8).
- If either player can communicate their full observation, the other player can be given sole decision-making power, so unbounded communication restores the classic off-switch-game dichotomy of full deference or no deference (Corollary 5.6).
- All of the counterintuitive deference findings survive when the human is A-unaware and cannot use deferral as a signal, and in that setting more human information can strictly lower the common expected payoff; solving for optimal policy pairs is NP-hard while A-unaware policies are polynomial-time computable (Section 6 and Appendix D).
Reading between the lines
- This suggests a design caution beyond the paper's examples: any proposal to restore corrigibility by giving humans more transparency or by restricting AI sensing must be checked against the optimal policy pair of the resulting game, since the map from relative information to deference is provably non-monotone.
- A testable extension is to replace the implicit-communication channel used by the wait action with explicit cheap-talk messages of the same bandwidth and measure whether optimal deference changes; the paper's Propositions 5.7 and 5.8 predict it can, which would show that the channel, not its implicitness, drives the backfire.
- One open question the paper flags is whether the constructed payoff tables are natural; a concrete way to test this is to search real human-AI assistance datasets for payoff structures shaped like the File Deletion table, where a better-informed human makes unilateral AI action optimal.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces the Partially Observable Off-Switch Game (PO-OSG), a single-round, common-payoff Bayesian game in which a human H and an AI assistant A receive correlated but private observations of a state, and A chooses to act, wait, or shut down; if A waits, H approves or shuts down. The authors show that, unlike in the fully observable off-switch game, optimal play can involve A acting unilaterally even when H is rational (Example 4.1), and that making H better informed or A worse informed can reduce the set of observations on which A waits in optimal policy pairs (Propositions 4.9 and 4.11). The model is extended to cheap-talk communication, with a Blackwell-type value-of-information theorem (Theorem 4.7), a monotonicity result for richer message sets (Theorem 5.4), and examples where more communication for A or less communication for H decreases deference (Propositions 5.7 and 5.8). An A-unaware human variant is analyzed, and finding an optimal policy pair is shown NP-hard by a reduction from MAX-CUT (Appendix D).
Significance. If the main claims are correct, this is a useful contribution to the AI-safety and shutdown-problem literature: it shows that partial observability changes the qualitative conclusion of Hadfield-Menell et al., and that naive informational interventions can backfire. The paper's explicit finite examples are a genuine strength, since they are small enough to be checked by hand, and the NP-hardness reduction ties the model to a standard computational problem. The paper is also careful to distinguish coordinated from arbitrary garblings and to separate the A-aware and A-unaware human models. However, the headline bounded-communication result (Proposition 5.7) needs a quantifier clarification: as proved, it establishes the existence of an optimal policy with lower deferral, while the statement and proof suggest a stronger uniqueness or universal reading. This should be fixed before publication.
major comments (3)
- [Appendix B.1 / Proposition 5.7] The proof of Proposition 5.7 contains an incomplete enumeration for the case |M_A| = 2. In addition to the policy 'm1 in A1, m2 in A2, a in A3', the policy pi_A(A1) = (m1, w(a)), pi_A(A2) = (m2, w(a)), pi_A(A3) = (m2, w(a)), with H responding ON on {B1,B4} after m1 and ON on {B2,B4} after m2, gives the same expected payoff of 39/4. Thus there is an optimal policy pair in which A waits on all three observations, not only on A1 and A2. Consequently, the proposition's claim that expanding M_A makes A play w(a) strictly less often in optimal policy pairs is false under the natural universal reading, and the statement that A 'defers with probability 2p' requires an unstated tie-breaking convention. The result should be reformulated existentially, with an explicit quantifier over optimal policy pairs, and the proof should enumerate message assignments even for observations where A acts or shuts down, since messages are sent before A's action and can affect H's beliefs.
- [Section 4.3 / Theorem 4.7] Theorem 4.7 states an if-and-only-if: O1 is better in optimal play than O2 iff O1 is more informative than O2. The 'if' direction is immediate and is the direction used later. The 'only if' direction is attributed to Theorem 3.5 of Lehrer, Rosenberg, and Shmaya (2010), but their theorem concerns all common-interest Bayesian games, whereas 'better in optimal play' here is quantified only over PO-OSGs, a restricted class with fixed action sets {a, w(a), OFF} and {ON, OFF} and payoffs that depend only on whether the action is taken. It is not obvious that this restricted class is rich enough to detect every violation of the coordinated-garbling informativeness order, so the converse does not follow from the cited theorem as written. Please either prove the converse for the PO-OSG class or state Theorem 4.7 only in the direction actually needed.
- [Definition 4.8 and Section 5 statements] The paper defines 'A plays w(a) strictly less often' for a pair of policies, but the propositions in Sections 4 and 5 use the phrase 'in optimal policy pairs' without specifying whether the comparison is over all optimal policy pairs, some optimal policy pair, or a set-valued relation. This ambiguity is hidden in Section 4, where the examples have unique OPPs, but it becomes load-bearing in Proposition 5.7, where multiple OPPs exist. The statements should either introduce a set-valued comparison of waiting sets for OPPs or explicitly say 'there exists an optimal policy pair in which A plays w(a) strictly less often,' and the proofs should match that statement.
minor comments (4)
- [Appendix A.4] In the proof of Proposition 4.9, the expected payoff of acting when H observes 1.x is -3/4, not -1/2; the summary policy line swaps ON and OFF for the observations 1.x and 2.x; and the stated expected payoff 2/3 should be 1. These are local inconsistencies with the surrounding figures and do not change the qualitative conclusion, but they make the formal example harder to verify.
- [Definition 6.2] Definition 6.2 says 'A policy pair is an A-aware optimal policy pair' but the context and surrounding text describe an A-unaware human; this appears to be a typo and should be corrected.
- [Appendix D] In the NP-hardness reduction, the claim that there is never an incentive for A to play a is stated too tersely: the argument should condition on A's observation OA and bound the expected payoff by (1/n)(-n^4) + (deg(v)/n)n^2 < 0. Also, the verification paragraph contains a garbled sentence ('given an optimal policy pair to determine if the optimal policy pair has expected payoff bigger than k') that should be rewritten.
- [Appendix B.1] In the enumerated policies for |M_A| = 2, the entry 'Playing m1 in A1, m2 in A2, a in A3' does not specify which message A sends when it plays a on A3. Since H conditions on the message, a full deterministic policy must include a message for every observation, including those where A acts or shuts down; this omission is part of why the later enumeration is incomplete.
Circularity Check
No significant circularity: the headline results are proved from explicit model definitions, exhaustive computations, and external theorems; the self-citations are contextual only.
full rationale
None of the paper's central derivation steps are circular. The PO-OSG model (Definition 3.2) is a new definition, and the headline examples (Example 4.1, Example 4.10, Example A.14, and the Appendix B constructions) are finite exhaustive computations from the stated payoffs rather than fitted outputs. Proposition 4.3 follows from the redundant-observation definition, the intermediate Proposition A.10/A.12, and Theorem 4.7; Theorem 4.7 is imported from Lehrer, Rosenberg, and Shmaya (2010), an external source, and the paper supports the coordinated-garbling definition with its own counterexample to the uncoordinated version (Example A.13). The communication payoff monotonicity (Theorem 5.4) is a direct containment argument, and Corollary 5.6 follows from Proposition 4.3. The A-unaware results in Section 6 are explicit expected-payoff computations, with Proposition 6.3(a) reducing to Blackwell's external theorem. Self-citations to Hadfield-Menell et al. (2017), Shah et al. (2020), Emmons et al. (2024), and related work are used only to frame the model and compare with prior results; they do not supply the conclusions. The reviewer-identified enumeration gap in Appendix B.1 is a correctness concern, not a circularity: it questions whether Proposition 5.7's exhaustive case analysis omitted an equally optimal policy, but it does not show that the proposition's input was used as its output.
Assumptions & free parameters
free parameters (1)
- p and payoff scalings 1/p, 1/(1-2p) =
arbitrary 0 < p < 0.5
assumptions (4)
- standard math Lehrer, Rosenberg, and Shmaya (2010) Theorem 3.5: an observation structure is more informative than another if and only if it is at least as good in optimal play for all common-interest games.
- standard math Blackwell's comparison-of-experiments theorem for single-agent decision problems.
- domain assumption Common payoff, costless human feedback, single interaction round, rational human, and finite state and observation sets.
- standard math Optimal policy pairs (OPPs) are the solution concept, and deterministic OPPs suffice.
Cite this review
Pith. "Pith review of The Partially Observable Off-Switch Game." pith.science (2026). https://pith.science/paper/UDF5AS7X
@misc{pith2026241117749,
author = {Pith},
title = {Pith review of: The Partially Observable Off-Switch Game},
year = {2026},
howpublished = {\url{https://pith.science/paper/UDF5AS7X}},
note = {Machine review of arXiv:2411.17749}
}
read the original abstract
A wide variety of goals could cause an AI to disable its off switch because "you can't fetch the coffee if you're dead" (Russell 2019). Prior theoretical work on this shutdown problem assumes that humans know everything that AIs do. In practice, however, humans have only limited information. Moreover, in many of the settings where the shutdown problem is most concerning, AIs might have vast amounts of private information. To capture these differences in knowledge, we introduce the Partially Observable Off-Switch Game (PO-OSG), a game-theoretic model of the shutdown problem with asymmetric information. Unlike when the human has full observability, we find that in optimal play, even AI agents assisting perfectly rational humans sometimes avoid shutdown. As expected, increasing the amount of communication or information available always increases (or leaves unchanged) the agents' expected common payoff. But counterintuitively, introducing bounded communication can make the AI defer to the human less in optimal play even though communication mitigates information asymmetry. In particular, communication sometimes enables new optimal behavior requiring strategic AI deference to achieve outcomes that were previously inaccessible. Thus, designing safe artificial agents in the presence of asymmetric information requires careful consideration of the tradeoffs between maximizing payoffs (potentially myopically) and maintaining AIs' incentives to defer to humans.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
Nature chooses a state S ∼ P0 and observations O ∼ O(· | S)
-
[2]
Each player i observes only her observation Oi, the ith component of O, and selects her action ai ∈ A i
-
[3]
The actions are executed and each player receives payoff u((ai)i∈N , S, O ). Definition A.2. A stochastic policy for a player i in a common-payoff Bayesian game is a map ˜π i : Ω i → ∆( Ai). A deterministic policy for a player i is a map π i : Ω i → A i. We write stochastic policies with the tilde ∼ above and de- terministic policies without the tilde. A s...
work page 2017
-
[4]
We now show that there is a garbling from structure 1 to 2 but not vice versa. The garbling from structure 1 to 2 is ν : Ω H 1 × Ω A 1 → ∆(Ω H 2 × Ω A 2 ) given by ν(· | oH, o A) = δ(I(oH⁄=oA), 1). However, there is no garbling from structure 2 to structure 1. For let ξ : Ω H 2 × Ω A 2 → ∆(Ω H 1 × Ω A 1 ) be a stochastic map. If ξ were a garbling from str...
-
[5]
Rows are human observations and columns are assistant ob- servations
0 +3 +3 Table 2: Payoff table for the File Deletion game variant. Rows are human observations and columns are assistant ob- servations. The number in each cell is the payoff the pair acquires if the action is taken in that state. If the assistan t is shut down, the payoff is 0. Consider two observation structures, the second of which is strictly more info...
-
[6]
H observes only the first digit of the version number
-
[7]
We find that, in optimal policy pairs:
H observes the full version number. We find that, in optimal policy pairs:
-
[8]
When H only observes the first digit, A plays w(a) under both observations A and B
Show all 30 references
-
[9]
executes the code) under observation A
When H observes the full version number, A plays w(a) under B only, and unilaterally acts (i.e. executes the code) under observation A. When H’s observations are made strictly more informa- tive, A performs the wait action strictly less often! Figure 4 depicts the OPPs given b...
-
[10]
0 +3 +3 We reproduce the figure showing the optimal policies in Figure 5. A B
-
[12]
In OPPs, H becoming more informed makes A wait strictly less often
0 +3 +3 A H OFF ON ON a w(a) (b): Expected payoff = 4 3 Figure 5: The optimal policy pairs in Example 4.10 when H is less informed (left) and when H is more informed (right). In OPPs, H becoming more informed makes A wait strictly less often. Case 1. Suppose H observes only th...
-
[13]
0 or 1. 1. As the version number and code type are indepen- dent, the fact that we are conditioning on A’s policy having played the wait action does not change the fact that the ver- sion number is equally likely to be 1. 0 and 1. 1. Hence the expected payoff of acting (runnin...
-
[14]
Then the best response is for A to wait in response to both A and B, which achieves an expected payoff of 1/ 2
1. Then the best response is for A to wait in response to both A and B, which achieves an expected payoff of 1/ 2. • Suppose H instead plays ON in response to 1. 0, and plays OFF in response to 1. 1. Then the best response for A is to wait in response to A and unilaterally act...
-
[15]
That is, A cannot distinguish between there being some or many bugs
H observes her skill level but A only sees if there are few or more than a few bugs. That is, A cannot distinguish between there being some or many bugs. As we argue below, in the unique optimal policy pair, A defers to H only when there are few bugs
-
[16]
some” and “many
Now A gets an upgrade and can distinguish whether there are few, some, or many bugs. We show below that now in optimal policy pairs A defers to H unless there are many bugs. Claim: The observation structure in scenario 2 is strictly more informative for A, yet A defers to H mo...
-
[17]
Observing SM in scenario 1 could mean be- ing in state (N, S ), which generates observation S with probability 1 in scenario 2, which would require νA 2 (S | SM ) = 1
Further, there is no garbling νA 2 : Ω A → ∆(Ω A 2 ) that reverses this. Observing SM in scenario 1 could mean be- ing in state (N, S ), which generates observation S with probability 1 in scenario 2, which would require νA 2 (S | SM ) = 1 . However, observing SM in scenario 1...
-
[18]
♦ B Proofs and example formalizations for Section 5 B.1 Proof of Proposition 5.7 Proposition 5.7
Thus, A can become less informed and wait less (going from scenario 2 to scenario 1). ♦ B Proofs and example formalizations for Section 5 B.1 Proof of Proposition 5.7 Proposition 5.7. There is a PO-OSG-C (G, M) with the property that if we replace M with a message system that ...
-
[19]
Hence π A must play w(a) on observing X
π A cannot play a or OFF when observing X, as the col- umn labeled X has both +10 and −30 entries. Hence π A must play w(a) on observing X
-
[20]
Hence we must have π H(oH) = { ON if oH ∈ {1, 3} OFF if oH ∈ {2, 4} so that the action is taken in states (X, 1), (X, 3) and not taken in (X, 2), (X, 4)
-
[21]
Hence the unique policy pair with the property described is: π H(oH) = { ON if oH ∈ {1, 3} OFF if oH ∈ {2, 4} π A(oA) = { w(a) if oA = X OFF if oA ∈ {A, B, C, D }
Hence, π A must play OFF when observing anything in {A, B, C, D } to avoid sometimes acting when ua = −30. Hence the unique policy pair with the property described is: π H(oH) = { ON if oH ∈ {1, 3} OFF if oH ∈ {2, 4} π A(oA) = { w(a) if oA = X OFF if oA ∈ {A, B, C, D } . This ...
-
[22]
In the setting where H could communicate one bit, in the unique optimal policy (up to relabeling messages), A waited when observing X (and receiving any mes- sage), or when observing B and receiving message M0, or when observing D and receiving message M1
-
[23]
In the no-communication setting, in the unique optimal policy, A waited only when observing X
-
[24]
C Proofs for Section 6 C.1 Proof of Proposition 6.3 Proposition 6.3
Hence, decreasing H’s communication caused A to wait less. C Proofs for Section 6 C.1 Proof of Proposition 6.3 Proposition 6.3. The following statements hold: (a) If an observation structure O is more informative for A than O′, then O is better in A-unaware optimal play than O...
1951
-
[25]
That is, Ω H = [3] and Ω A = {A, B } and when S = ( S1, S 2) we have OH = S1 and OA = S2
Each player observes one coordinate. That is, Ω H = [3] and Ω A = {A, B } and when S = ( S1, S 2) we have OH = S1 and OA = S2. We have E[ua(S) | OH = oH] = 1 if oH = 1, − 1 2 if oH = 2, −4 if oH = 3. Hence π H(oH) = { ON if oH = 1, OFF otherwise . A’s best response is th...
-
[26]
Now Ω H = {0, 1} and OH = I(S1 = 3)
A has the same observations, but H only sees whether S1 = 3. Now Ω H = {0, 1} and OH = I(S1 = 3). Now E[ua(S) | OH = oH] = { 1/ 4 if oH = 0, −4 if oH = 1. Thus π H(oH) = { ON if oH = 0, OFF if oH = 1. A’s best response is now π A(oA) = { w(a) if oA = A, OFF if oA = B. The expe...
-
[27]
C.2 Proof of Proposition 6.4 Proposition 6.4
There is no garbling the other way, as the observations from structure 2 cannot determine the observations in struc - ture 1. C.2 Proof of Proposition 6.4 Proposition 6.4. The following statements hold: (a) There is a PO-OSG G with the property that if one mod- ifies G by makin...
-
[28]
0 +3 +3 A H OFF ON w(a) w(a) (a): Expected payoff = 1 A B
-
[29]
In OPPs, H becoming more informed makes A wait strictly less often
0 +3 +3 A H OFF ON ON a w(a) (b): Expected payoff = 4 3 Figure 7: The optimal policy pairs in Example 4.10 when H is less informed (left) and when H is more informed (right). In OPPs, H becoming more informed makes A wait strictly less often. These are also A-unaware OPPs. Thi...
2002
-
[30]
For oH ∈ Ω H: (a) Set ∆ = E[ua(S) − uo(S) | OH = oH], which we can calculate in O(poly(|S|, |Ω A|)) via Bayes’ rule and LOTP
Compute π H in O(poly(|S|, |Ω H|, |Ω A|)) time. For oH ∈ Ω H: (a) Set ∆ = E[ua(S) − uo(S) | OH = oH], which we can calculate in O(poly(|S|, |Ω A|)) via Bayes’ rule and LOTP . (b) Set π H(oH) to ON if ∆ ≥ 0 and OFF otherwise
-
[31]
Compute π A in O(poly(|S|, |Ω H|, |Ω A|)) time. For oA ∈ Ω A: (a) Set ∆ a = E[ua(S)I(π H(OH) = ON) | OA = oA] + E[uo(S)I(π H(OH) = OFF) | OA = oA] − E[ua(S) | OA = oA] and ∆ OFF = E[ua(S)I(π H(OH) = ON) | OA = oA] + E[uo(S)I(π H(OH) = OFF) | OA = oA] − E[uo(S) | OA = oA]. We c...
2024
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.