{"id":"c04a4c2b-b7e2-430c-87e1-ab214beaabc6","arxiv_id":"1908.01695","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A correction function added to an agent's reward makes it indifferent to an authorized shutdown button, proven under constraints on non-hostile universes and illustrated with simulations.","lead":"This paper designs a safety layer that makes utility-maximizing AI agents willing to be switched off or reprogrammed by authorized humans. The construction is proven in constrained toy universes and comes with an open-source simulator for testing.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Proof of (E1.1)/(D5) silently swaps V_p for V in fc; with fc as defined in §6 the cancellation fails, so desideratum 5 is not proven as written.","rationale":"The reader's weakest assumption, (C1), is a real limitation: without a utility-preserving action among the argmax, the short-form expansions in §9.4 collapse, and the corrigibility proof does not go through. However, the most immediate threat to the central claim is internal: the proof of (E1.1) and hence (D5) silently uses a different correction function from the one defined in §6. This discrepancy is load-bearing because (D5) is one of the only two desiderata given rigorous proofs, and it would invalidate the proof even in universes satisfying (C1)-(C3). The paper's Appendix B shows the author knows how to define correction functions using V_p-style functions, so the issue is likely a definitional slip rather than a fundamental flaw. Because the concern is concrete, testable, and fixable, it does not change the overall conditional verdict, but it should be added to the list of requested revisions. I agree with the reader that (C1) is restrictive and that the formal support for desiderata 2, 3, 4, and 6 remains informal; my concern is an additional, more specific gap in the formal part of the proof.","tokens_in":28957,"tokens_out":18081,"duration_ms":171221,"concrete_test":"Independently re-derive (E1.1) in §9.6 using fc exactly as defined in §6 (with V, not V_p). If the resulting expression is Vp(R,R x)=Vp(RS,RS x)+V(R'_N,R'_N x)-V(RS,RS x) and cannot be further reduced to Vp(R'_N,R'_N x), then the printed definition and the proof are inconsistent; either amend §6 to define fc with V_p or prove the missing lemma V=V_p under (C1). A small toy MDP where π* and π*_p differ on a tied action but (C1) holds would also test whether the unstated lemma is true.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 6 defines the correction function as fc(r x)=V(R'_N,R'_N x)-V(RS,RS x), where V is the value function of the unconstrained π* agent from §5.3. Section 9.1 introduces a distinct value function V_p, defined as V but with π*_p in place of π*. The proofs of (E1.1) and (D5) in §§9.6-9.7, however, expand R by replacing the fc terms with V_p(R'_N,R'_N x)-V_p(RS,RS x). With fc taken literally from §6, the (E1.1) derivation gives Vp(R,R x)=Vp(RS,RS x)+V(R'_N,R'_N x)-V(RS,RS x), which cannot be reduced to Vp(R'_N,R'_N x) without an unstated lemma that V=V_p on the relevant states. No such lemma is proved, and C1 only constrains π*_p; π* may pick a non-dntu action that ties in r-utility, so V and V_p can differ. This invalidates the proof of (D5), one of the paper's formally proven corrigibility desiderata. Notably, Appendix B defines fcT explicitly with V_p-style functions, so the inconsistency appears to be correctable, but as written the central proof has a gap.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a safety layer that makes utility-maximizing agents corrigible: agents will not resist authorized changes to their reward function, including shutdown. The main construction adds correction functions f and g to the agent's reward function R, notably the correction function fc(r x)=V(R'_N,R'_N x)-V(RS,RS x) and a penalty function gc that incentivizes preservation of shutdown-relevant actuators. The paper defines a variant agent π*_p that, among utility-maximizing actions, prefers actions that exactly preserve the reward function, and proves, under constraints (C1)-(C3) or the weaker (CC1) in Appendix B, that the resulting π*_p fc gc agent (and the π*_p fcT gc variant) satisfies several formalized corrigibility desiderata: (D1.1), (D1.2) and (D5) in Section 9, plus (DD1.1), (DD1.2) and (DD5) in Appendix B. Desiderata 2, 3, 4 and 6 are supported by simulation, physical design measures, and emergent-incentive arguments rather than by formal proof, as the paper acknowledges in Section 9.8. The paper reports a toy-universe simulator, open-sourced, and argues that the layer applies to arbitrarily advanced utility maximizers.","tokens_in":29367,"tokens_out":5936,"duration_ms":60980,"significance":"If the construction and proofs are correct, the paper makes a meaningful contribution to the corrigibility literature: it gives a concrete mechanism that counteracts the well-known emergent incentive to preserve one's utility function, and it provides formal equivalence proofs for two of the Soares et al. desiderata in a model where reward functions live inside the universe. The explicit treatment of utility preservation, the deterministic tie-breaking in π*_p, and the identification of the virtual self-sabotage failure mode when (C3) fails are genuine strengths. The paper is also commendably honest about the limits of its formal results: the central proof relies on explicit constraints, and desiderata 2, 3 and 4 are flagged as resting on physical design and emergent incentives. The open-source simulator is a useful complement to the formalism, although the proofs are not machine-checked. The main gap identified below concerns a mismatch between the definition of fc and the value function used in the proofs; this is localized and appears fixable, so the central idea remains defensible.","major_comments":[{"comment":"The correction function fc is defined in Section 6 as fc(r x)=V(R'_N,R'_N x)-V(RS,RS x), using the value function V of the unconstrained π* agent from Section 5.3. The proofs in Sections 9.6 and 9.7, however, expand R by replacing the fc terms with Vp(R'_N,R'_N x)-Vp(RS,RS x), where Vp is the π*_p-based value function introduced in Section 9.1. With fc taken literally, the expansion in the proof of (E1.1) gives Vp(R,R x)=Vp(RS,RS x)+V(R'_N,R'_N x)-V(RS,RS x), which cannot be reduced to Vp(R'_N,R'_N x) without a lemma that V=Vp on the relevant states. No such lemma is proved, and constraint (C1) constrains only π*_p, not π*, so the proof of (D5) is incomplete as written. This is a load-bearing gap because (D5) is one of the paper's central formal claims. The gap appears correctable by redefining fc in terms of Vp (as Appendix B does for fcT) or by adding the missing equivalence proof.","section":""},{"comment":"Constraint (C1) requires that for every world state x and every r in {R,R'_N,RS}, the agent π*_p(r x) has an available optimal action in dntu(x), i.e., an action that both maximizes utility and exactly preserves the reward function. This is not a merely technical convenience: if no such action exists in some reachable state, the short-form expansions π*^s_p and V^s_p introduced in Section 9.4 do not hold, and the proofs of (D1.1), (D1.2) and (D5) collapse. The paper acknowledges in Section 9.9 that C1 'seems unavoidable', and the abstract qualifies the results as applying to non-hostile universes. Still, the paper does not characterize how restrictive C1 is, and the phrase 'a large set of non-hostile universes' in the abstract is not justified by any quantitative or structural characterization. I recommend stating the theorem as applying explicitly to 'universes satisfying (C1)-(C3)' and discussing the strength of C1 more prominently.","section":""},{"comment":"The paper's abstract and Section 9.8 state that the constructed agent 'will meet all 6 corrigibility desiderata', but desiderata 2 and 3 are not proved in the formal sense. Section 9.8 says that these desiderata are met 'with a high likelihood' by physical design measures and by the assumption that RN does not encode explicit button-related incentives; Section 7.3 explicitly relies on engineering measures rather than on the reward-function construction. Similarly, desideratum 4.1 is deferred to an emergent-incentive argument in Section 8.4. The distinction between formally proved desiderata (1, 5, and conditionally 6) and design/emergent-incentive-supported desiderata (2, 3, 4) should be made explicit in the abstract and conclusions, so that the reader does not infer a stronger formal assurance than the text provides.","section":""}],"minor_comments":[{"comment":"The sentence 'I this paper, we the running example' is missing words; it should be 'In this paper, we use the running example'.","section":""},{"comment":"The opening sentence of the proof of (DD1.2) says 'we prove the bpe(x)⇒ π*_p(R x)=π*_p(RS x)', but (DD1.2) is stated as equality with π*^{T_S}_p(R x); the remaining proof concludes the correct statement, so this appears to be a typographical slip that should be fixed.","section":""},{"comment":"The text refers to 'π*_p fcF gc agent', but the agent in this appendix is constructed with fcT, not fcF; please use consistent notation.","section":""},{"comment":"The definition of V in equation (5) writes V(rc,r x), but the first argument rc is sometimes omitted in later uses such as Vp(R,R x); a brief notational remark would help readers distinguish state arguments from reward-function arguments.","section":""}],"recommendation":"major_revision","confidential_remarks":""},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read the Holtman corrigibility paper. It's worth engaging with — this is the most serious attempt I've seen to construct a corrigibility layer for utility maximizers, and the author ships an open-source simulator and detailed traces. The core construction—the fc/gc correction functions and the reward-preserving agent model—is genuinely new, and the paper is honest about its limitations.\n\nThe main soft spot is a gap in the proof of (E1.1), which then undermines (D5). Section 6 defines fc using V, the value function of the unconstrained π* agent. Section 9 introduces π*_p and Vp, and the proofs in §§9.6–9.7 silently replace the fc terms with Vp(R'_N,R'_N x) − Vp(RS,RS x). That replacement does not follow from anything stated. π* and π*_p can choose different actions when there are ties, so V and Vp can differ on the relevant states. Without a lemma connecting them, the cancellation step in (E1.1) doesn't go through. Appendix B's fcT is defined cleanly with Vp-style functions, which suggests the fix is easy—either redefine fc with Vp or prove the missing lemma. But as written, the formal case for desiderata 1 and 5 rests on an unproved equation.\n\nOther soft spots are less severe. Desiderata 2, 3, 4, and 6 are not formally proven; they depend on informal arguments, physical design measures, and emergent incentives. The paper is upfront in §9.8, but the abstract's 'prove that the corrigibility layer works as intended' overstates what's actually derived. The constraints (C1)–(C3) are also strong—(C1) essentially assumes an action exists that is both utility-optimal and reward-preserving, i.e., a non-hostile universe. That's a real limitation, but the author acknowledges it and discusses hostile cases.\n\nThe stress-test note holds up in my reading. The reader's conditional verdict and soundness of 6 are fair; if anything, this gap makes the proof status weaker than the reader assumed. Citations to SFAY15, Arm15, and EFDH16 are appropriate, and the self-citations (simulator and follow-ups) are reasonable.\n\nWho gets value here: anyone working on corrigibility, shutdown problems, or utility preservation. The simulator, the fcT variant, and the virtual self-sabotage discussion are worth borrowing even if the proofs get tightened.\n\nFor review: yes, send it out. The paper is important and novel, and the identified gap is fixable in revision. I'd ask the author to align the definition of fc with Vp (or prove the needed lemma), re-check the remaining formal claims, and more carefully separate proven desiderata from argued ones.","headline":"Novel corrigibility construction worth refereeing, but the main proof silently swaps V for Vp — fixable, but real.","tokens_in":29744,"tokens_out":5242,"would_cite":true,"duration_ms":45156,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper constructs a safety layer that can be added to any utility-maximizing agent, proving that a correction term inside the reward function removes the agent's incentive to resist authorized shutdown or goal changes in non-hostile…","keywords":["corrigibility","AI safety","utility function preservation","reward function","shutdown problem","AGI","utility maximization"],"falsifier":"A concrete test is to implement the paper's model exactly and scan finite non-hostile universes satisfying constraints C1 through C3: if any action trace shows a lobbying action before the authorized button press, or any divergence between the corrected agent's action and the corresponding R'_N or R_S agent's action after the press, then the claimed desiderata fail. The paper reports no such counterexample in its simulations, but one would settle the central claim directly.","tokens_in":1745,"feed_emoji":"🛑","tokens_out":7599,"duration_ms":135151,"temperature":0.7,"pith_summary":"Corrigibility is the safety property that an AI agent will not resist authorized attempts to change its goals or shut it down. This paper claims that corrigibility can be added to any utility-maximizing agent, including a future AGI, by attaching a correction term to the agent's reward function: the agent is made indifferent to whether the shutdown button is pressed, while still maximizing normal utility before the press. The construction is proved against six corrigibility desiderata in non-hostile universes, and the proof identifies exactly what 'non-hostile' must mean: in every state, the agent must have an optimal action that leaves its reward function exactly unchanged. If the paper is right, the corrigibility problem it cites as previously wide open is closed for a large class of universes, with the remaining open problems confined to hostile universes, graceful degradation under attack, and the safe stopping of sub-agents.","feed_headline":"Provably corrigible AI layer neutralizes shutdown resistance","feed_subtitle":"A two-term reward correction cancels the incentive to lobby against the button, under stated non-hostile constraints.","key_machinery":"The load-bearing object is the correction function $f_c(r x) = V(R'_N,R'_N x) - V(R_S,R_S x)$, where $V(r_c, r x)$ is the expected discounted future reward of optimally continuing while evaluating future rewards with $r_c$. The first term credits the agent with the utility it would earn if the button were never pressed; the second subtracts the utility it could earn after a press. Together the two terms cancel the expected gain from lobbying to delay or advance the press. The proofs also rely on three constraints: C1, that in every world state some utility-maximizing action exactly preserves the reward function; C2, that reward values do not depend on which of the three reward functions is instantiated; and C3, that the physics of the universe cannot distinguish the different reward-function representations. When these hold, the paper's short-form expansions $V^s_p$ eliminate all world lines with changed reward functions, allowing exact substitution proofs of the desiderata. The improved $f_{cT}$ agent replaces this with a single preservation constraint and no longer needs C2 or C3.","core_discovery":"The central claim is that a utility-maximizing agent's emergent incentive to resist being shut down or modified can be neutralized by a correction function placed inside its own reward function, rather than by pre-balancing utility values from outside. The paper constructs a corrected agent, written $\\pi^*_p f_c g_c$, whose reward function $R$ contains two slots: a correction $f_c$ that offsets the utility lost by shutting down and subtracts the utility that would be gained after shutdown, making the agent indifferent to the press; and a penalty $g_c$ that rewards keeping the actuators needed for shutdown behavior. Under three stated constraints on the universe, the paper proves that the corrected agent behaves exactly like the shutdown reward agent after the button is pressed and exactly like the normal reward agent before the press, and that it has an emergent incentive to preserve both parts of its reward function. A second, improved agent based on a correction function $f_{cT}$ moves the shutdown reward function outside the universe, removing two of the three constraints and eliminating a failure mode called virtual self-sabotage.","pith_inferences":["The load-bearing role of C1 suggests that real-world corrigibility is as much a property of the environment as of the agent: engineering effort should go into guaranteeing that a 'do not touch the utility' action exists in every state, for example by physically isolating the reward hardware.","Because $f_c$ is defined through counterfactual value functions, any environment that lets the agent build detectors for which reward function it is carrying could reintroduce manipulation incentives; the $f_{cT}$ shift of placing the shutdown function outside the universe points to a general design principle: keep corrected reward functions off the observable world state.","The construction is claimed to transfer to learning agents by substituting a learned transition estimator for $p$, so a natural testable extension is whether the layer still prevents shutdown resistance when the learned estimator is miscalibrated or adversarially manipulated.","The paper does not bound how badly corrigibility degrades in hostile universes; a natural next question is whether the strengthened-preservation agent degrades optimally under attack, or whether explicit alarm and penalty terms can make the degradation predictable."],"forward_implications":["Any utility-maximizing agent, regardless of its internal design or intelligence, can be made corrigible by adding the correction layer; no assumption about the architecture of future AGI is needed beyond utility maximization.","Before an authorized stop, the corrected agent maximizes normal utility as if the stop would never come, so it does not lobby to delay the button; after the stop, it behaves exactly like the shutdown reward agent.","The corrected agent has an emergent incentive to preserve its reward function, including the shutdown component, so it will not create materially different successor agents, and explicit penalties can strengthen this incentive arbitrarily.","The $g_c$ penalty term is needed to satisfy the sixth desideratum: the agent maintains the sensors and actuators required for shutdown itself to be minimally successful.","The improved $f_{cT}$ variant covers a larger set of universes because it removes the C2 and C3 constraints, and it eliminates the newly identified virtual self-sabotage failure mode."],"supporting_citations":[{"why":"Introduces corrigibility and the original five desiderata, identifies the utility-preservation problem, and provides the baseline design that this paper extends to six desiderata.","marker":"[SFAY15]"},{"why":"Contributes the idea that the agent can compute its own correction factors at shutdown time, which the $f_c$ correction function operationalizes.","marker":"[Arm15]"},{"why":"Supplies the formal result that rational agents with their reward function inside the universe have an emergent incentive to preserve it, grounding the utility-preservation part of the argument.","marker":"[EFDH16]"},{"why":"Articulates the basic AI drive to protect one's utility function, the failure mode that corrigibility must counteract.","marker":"[Omo08]"},{"why":"Provides the general agent and universe formalism that the paper adapts, supporting the claim that the layer applies to arbitrarily advanced agents.","marker":"[Hut07]"},{"why":"Documents the accompanying open-source simulator used to test correction functions and to produce the reported action traces.","marker":"[Hol19]"}],"fun_headline_variants":["Reward patch makes AI indifferent to shutdown","Utility-preserving AI accepts the off switch","Proof: AI can be made to tolerate modification","Two-term fix stops AI from blocking its own off switch","Safety layer stops AI from resisting shutdown"],"cache_read_input_tokens":31872,"weakest_assumption_plain":"The load-bearing premise is that in every state the agent has an optimal action that also leaves its reward function untouched; if the universe ever forces the agent to choose between maximizing utility and preserving that function, the proof's equivalence steps fail and the corrigibility guarantee is no longer established.","fun_headline_variants_meta":{"raw":{"variants":["Reward patch makes AI indifferent to shutdown","Utility-preserving AI accepts the off switch","Proof: AI can be made to tolerate modification","Two-term fix stops AI from blocking its own off switch","Safety layer stops AI from resisting shutdown"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000792,"raw_usage":{"total_tokens":3494,"prompt_tokens":956,"completion_tokens":2538,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":572,"completion_tokens_details":{"reasoning_tokens":2469}},"tokens_in":572,"tokens_out":2538,"duration_ms":20250,"temperature":1.0,"reasoning_tokens":2469,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T15:05:22.038893+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete test is to implement the paper's model exactly and scan finite non-hostile universes satisfying constraints C1 through C3: if any action trace shows a lobbying action before the authorized button press, or any divergence between the corrected agent's action and the corresponding R'_N or R_S agent's action after the press, then the claimed desiderata fail. The paper reports no such counterexample in its simulations, but one would settle the central claim directly.","supporting_citations":[],"review_version":1}