REVIEW 2 major objections 5 minor 1 cited by
The Policy Cliff: A Theoretical Analysis of Reward-Policy Maps in Large Language Models
T0 review · 2 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read The reward-to-policy map in reinforcement learning is generically discontinuous at any reward with non-unique optimal actions, so sudden, hard-to-predict policy jumps in LLMs are a mathematical consequence of exact RL rather than a…
desk verdict Solid single-reward discontinuity theory worth publishing; the multi-reward extension overclaims and the empirical section is weak, but the core deserves a serious referee. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the reward-policy map $\mathcal{M}_{\mathrm{RL}}: R \mapsto \pi^*_R$, analyzed as three stages: $R\mapsto Q^*_R$ is Lipschitz with constant $1/(1-\gamma)$ (Proposition 2.4); $R\mapsto A^*(s;R)=\arg\max_{a} Q^*_R(s,a)$ is upper hemi-continuous but not lower hemi-continuous (Lemma 2.5); and the selection from $A^*$ to a policy is where continuity can fail. The discontinuity proof uses a continuous bump $\varphi$ concentrated near $(s_0,a_2)$, forms $Q_\varepsilon=Q_0+\varepsilon\varphi$, then defines $R_\varepsilon$ by inverting the Bellman operator, yielding $\|R_\varepsilon-R_0\|_\infty \le \varepsilon(1+\gamma)$ while making $a_2$ uniquely optimal. In the multi-reward setting the same machinery applies to an effective reward $R_{\mathrm{eff}}(s,a;\mathbf{R})=\sum_k w_k(s)R_k(s,a)$ with fixed state-dependent weights; entropy regularization replaces the argmax with a softmax, making the policy map Lipschitz with constant $1/(2\alpha(1-\gamma))$.
What would settle it
In a tabular MDP where two actions have exactly equal Q-values at a state, apply the paper's inverse-Bellman perturbation $\varepsilon\varphi$ to make one action uniquely optimal; if for arbitrarily small $\varepsilon$ the selected optimal action does not switch from $a_1$ to $a_2$, the discontinuity proposition fails. On real LLMs, train with reward $R_0$ and $R_0+\delta$ for a sequence of very small $\delta$ and measure the distribution of generated first tokens; if behavior changes smoothly with $\delta$ rather than jumping at a degeneracy, the generic-discontinuity claim is falsified in that regime.
Extended reading notes
Core claim
The discovery is a pair of discontinuity theorems (Propositions 2.9 and 2.10). Suppose at some state $s_0$ the optimal Q-function for reward $R_0$ has a finite optimal-action set containing at least two distinct actions, and the policy selection rule picks $a_1$ at $s_0$. Then for every $\varepsilon>0$ there exists a reward $R_\varepsilon$ with $\|R_\varepsilon-R_0\|_\infty \le \varepsilon(1+\gamma)$ whose unique optimal action at $s_0$ is $a_2$; under the uniform-stochastic selection rule, the policy at $s_0$ jumps from a uniform distribution over $m\ge 2$ actions to a Dirac mass at $a_2$. The proof works in Q-space: perturb $Q_0$ by a bump at $(s_0,a_2)$ and invert the Bellman operator to recover a reward that synthesizes the switch. Because non-unique optima are unavoidable whenever multiple token sequences or reasoning traces receive the same reward, the reward-policy map is generically discontinuous in exactly the regime where LLM decoding is trained.
Load-bearing premise
The multi-reward results assume that the weights $w_k(s)$ used to combine specialized rewards into the effective reward are fixed and do not themselves depend on the rewards or the policy; if those weights are learned or shift with the reward tuple, Lemma 4.2 and Proposition 4.7 no longer apply, as the paper acknowledges.
Editorial extensions
If this is right
- Under incomplete rewards, an optimal policy for the training reward is strictly suboptimal for the true reward whenever a co-optimal action has positive missing-reward advantage (Proposition 3.1), so spurious reasoning and instruction disobedience are rational, not bugs.
- Small additive bonuses or penalties act as tie-breakers: a tiny perturbation can make a desired behavior uniquely optimal, explaining abrupt format or length control (Proposition 3.2).
- In multi-reward RL, stability is governed by the effective reward $R_{\mathrm{eff}} = \sum_k w_k(s) R_k(s,a)$; non-unique optima of $Q^*_{\mathrm{eff}}$ make the policy map discontinuous with respect to the reward tuple (Propositions 4.5 and 4.6).
- Entropy regularization restores Lipschitz continuity with $d_{TV}(\pi_1,\pi_2) \le \|R_1-R_2\|/(2\alpha(1-\gamma))$ (Proposition 4.7), so small reward changes yield small behavior changes, at the cost of increased stochasticity and some suboptimality.
- Sudden shifts such as the move from transparent cheating to obfuscated deception, and RLHF 'sophistry,' are explained as policy cliffs arising from degenerate optima; the direction of a reward perturbation matters more than its scale.
Reading between the lines
- An implication the paper leaves implicit is that empirical 'trade-offs' between capabilities may be cliff crossings rather than smooth curves: probing a reward neighborhood, rather than a single point, should reveal where behavior jumps.
- The inverse-Bellman construction is a constructive design tool: one can deliberately engineer reward perturbations that force a desired policy switch without changing optimal Q-values elsewhere, which could enable targeted behavior control in reasoning and alignment.
- Because the entropy-regularization Lipschitz constant grows as $1/\alpha$, adaptive temperature annealing offers a testable knob for trading stability against decisiveness during RL training.
- Whether real LLM training dynamics (PPO noise, finite samples) exhibit the exact discontinuities of the map, or smooth them out, is an open empirical question; the theory predicts these cliffs will appear as run-to-run variance and sharp behavioral jumps near degenerate rewards.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper formalizes the reward-policy map for infinite-horizon discounted MDPs and proves that the optimal Q-function is Lipschitz in the reward (Prop. 2.4), the argmax correspondence is upper hemi-continuous (Lemma 2.5), the deterministic policy map is continuous under local uniqueness (Theorem 2.7), and discontinuous at rewards with at least two optimal actions (Props. 2.9 and 2.10) via an inverse-Bellman bump construction. It applies these results to LLM text generation, interprets 'clever slacker' and tie-breaker phenomena (Props. 3.1 and 3.2), and extends the analysis to multi-reward training by defining a state-dependent effective reward Reff = sum_k w_k(s) R_k(s,a). It proves stability statements for Reff-optimal policies and Lipschitz continuity for entropy-regularized softmax policies (Prop. 4.7), then reviews empirical LLM findings and reports two perturbation experiments. The single-reward core appears sound. The multi-reward extension characterizes policies optimal for Reff rather than the global objective J(pi) in Eq. (6), a gap acknowledged in Remark 4.1 and Section 4.6.
Significance. If the main results hold, the paper makes a useful formal contribution: it gives a constructive proof that arbitrarily small sup-norm reward perturbations can flip optimal actions when the argmax is non-unique, and it gives a clean Lipschitz-continuity justification for entropy regularization with an explicit constant. The inverse-Bellman construction is elegant, and the paper is unusually candid about its assumptions and limitations. However, the multi-reward section is the main advertised extension for LLM practice, and it currently rests on an assumed fixed-weight aggregation mechanism that is not linked to the actual multi-reward objective J(pi). The empirical perturbation experiments are suggestive but do not test the effective-reward mechanism quantitatively. The single-reward part could stand as a self-contained theory of policy cliffs, while the multi-reward part needs either additional results or a substantial scoping revision.
major comments (2)
- [§4.2-4.6, Eqs. (6)-(7), Remark 4.1] The multi-reward analysis does not characterize the policy that optimizes the global objective J(pi) defined in Eq. (6). That objective is a mixture of episode-specific reward functions, and the paper instead analyzes the policy pi*_Reff that is optimal for the constructed effective reward Reff(s,a;R)=sum_k w_k(s)R_k(s,a) with fixed weights. Remark 4.1 explicitly concedes that pi*_Reff coincides with a maximizer of J only in the special case of identical initial distributions with w_k(s)=p_k, and Section 4.6 concedes that if the weights depend on R or on the policy, Lemma 4.2 and Proposition 4.7 no longer apply. Since the abstract and Section 4.5 present this as an extension to realistic multi-reward RL, the missing link between J and pi*_Reff is load-bearing. The paper should either prove a quantitative approximation result between maximizers of J and pi*_Reff under explicit assumptions, or reframe the multi-reward contribution as a stability analysis of a fixed-weight effective-reward model and adjust the abstract and conclusion accordingly.
- [§5.6 and Appendix B] The controlled experiments in Section 5.6 train on the actual multi-reward objective and report aggregate performance shifts, but they do not measure the effective reward Reff, do not verify that the learned policy is optimal for Reff with fixed weights, and do not test any quantitative prediction of the theory (such as the epsilon(1+gamma) perturbation bound or the (m-1)/m total-variation jump). They are therefore consistency evidence rather than validation of the effective-reward mechanism. In addition, Appendix B states that the work lacks systematic empirical validation designed to quantitatively test the theory's predictions, which is in tension with the abstract's claim of validation through perturbation experiments. The authors should align these statements and, if possible, report the actual perturbation norms and check for threshold behavior.
minor comments (5)
- [Appendix D, Table 2] The heading 'Sectuion' should be 'Section'.
- [§4.4] Section 4.4 references Figure 3 as a conceptual illustration, but the figure does not appear in the manuscript; include it or remove the reference.
- [§2.5, Props. 2.9-2.10] Propositions 2.9 and 2.10 assume A*(s0;R0) is finite; since Assumption 2.1 allows compact metric action spaces with infinite argmax sets, the theorem statements should state this restriction explicitly (the finite LLM vocabulary is not affected).
- [§3.1] Section 3.1 models generation as an infinite-horizon discounted MDP but does not specify the terminal or absorbing state or the treatment of trajectories truncated at Tmax; please clarify.
- [§5.6] Section 5.6 reports aggregate benchmark scores; please state explicitly that these are indirect indicators of policy-map discontinuity rather than direct observations of argmax jumps.
Circularity Check
Single-reward core is self-contained; the multi-reward extension is a disclosed modeling tautology rather than a hidden circularity.
-
self definitional
[Section 4.2, Eq. (7) and Remark 4.1; Section 4.6]
"The policy subject to our continuity analysis, denoted π∗_R_eff, is the optimal policy for the standard MDP defined by this Reff. ... The alignment of π∗_R_eff with π∗_R (the maximizer of J(π)) depends on how well wk(s) are chosen or learned. If, for instance, the initial state distributions were identical (Dk = Dinit for all k) and one chose wk(s) = pk (constant weights), then J(π) simplifies to ... and π∗_R_eff would indeed be π∗_R."
The multi-reward stability results are stated about π∗_R_eff, which the paper defines as the policy optimal for the constructed effective reward Reff(s,a;R) = Σ wk(s)Rk(s,a). Propositions 4.5 and 4.6 therefore establish continuity and discontinuity of the map R ↦ π∗_R_eff, i.e., of a policy that is optimal for Reff by definition. The paper's broader takeaway that stability in multi-reward RL is 'governed by an effective reward aggregation mechanism' is consequently a property of the object it introduced, not a theorem derived from the actual global objective J(π) in Eq. (6).
full rationale
The single-reward theory is self-contained and not circular. Proposition 2.4 is a standard Lipschitz bound on the Bellman operator; Lemma 2.5 is a direct application of Berge's Maximum Theorem; Propositions 2.9 and 2.10 construct a convergent sequence of rewards by inverting the Bellman equation and exhibit a genuine discontinuity of the argmax selection. No fitted parameter is renamed as a prediction, and no cited uniqueness theorem from the authors is used to force a conclusion. The entropy-regularization result in Proposition 4.7 is a standard soft-Bellman contraction plus a softmax Lipschitz bound, cited to Geist et al. (2019), and is derived rather than assumed. The only mild circularity is in the multi-reward extension: the paper defines an effective reward Reff, defines the policy under analysis as optimal for that Reff, and then proves that this policy's stability depends on Reff. This is a modeling tautology, but the authors explicitly flag the gap between Reff-optimality and maximization of the global objective J(π) in Remark 4.1 and Section 4.6. Because the core single-reward results and the entropy-regularization proof are independent mathematical derivations with stated assumptions that do not include the target conclusions, the overall circularity score is low. The paper is honest about the conditional nature of its multi-reward claims, so this is a 2 rather than a higher score.
Assumptions & free parameters
free parameters (1)
- aggregation weights w_k(s)
assumptions (5)
- standard math Assumptions 2.1-2.3: S and A are compact metric spaces, the transition kernel has the Feller property, and gamma < 1.
- domain assumption Finite-state and finite-action LLM token generation is an MDP with deterministic transitions and a fixed horizon or discounting structure (Section 3.1).
- ad hoc to paper Aggregation weights w_k(s) are fixed and independent of R and pi (Eq. 7 and Section 4.6).
- ad hoc to paper The policy optimized in multi-reward RL is represented as optimal for the effective reward R_eff rather than for the global objective J(pi) (Remark 4.1).
- domain assumption The discontinuity constructions require a finite set of optimal actions at the tie state (Propositions 2.9 and 2.10).
invented entities (1)
-
effective reward function R_eff(s,a;R) = sum_k w_k(s) R_k(s,a)
Cite this review
Pith. "Pith review of The Policy Cliff: A Theoretical Analysis of Reward-Policy Maps in Large Language Models." pith.science (2026). https://pith.science/paper/J7UZJOUZ
@misc{pith2026250720150,
author = {Pith},
title = {Pith review of: The Policy Cliff: A Theoretical Analysis of Reward-Policy Maps in Large Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/J7UZJOUZ}},
note = {Machine review of arXiv:2507.20150}
}
read the original abstract
Reinforcement learning (RL) plays a crucial role in shaping the behavior of large language and reasoning models (LLMs/LRMs). However, it often produces brittle and unstable policies, leading to critical failures such as spurious reasoning, deceptive alignment, and instruction disobedience that undermine the trustworthiness and safety of LLMs/LRMs. Currently, these issues lack a unified theoretical explanation and are typically addressed using ad-hoc heuristics. This paper presents a rigorous mathematical framework for analyzing the stability of the mapping from a reward function to the optimal policy. We show that policy brittleness often stems from non-unique optimal actions, a common occurrence when multiple valid traces exist in a reasoning task. This theoretical lens provides a unified explanation for a range of seemingly disparate failures, reframing them as rational outcomes of optimizing rewards that may be incomplete or noisy, especially in the presence of action degeneracy. We extend this analysis from the fundamental single-reward setting to the more realistic multi-reward RL across diverse domains, showing how stability is governed by an "effective reward" aggregation mechanism. We also prove that entropy regularization restores policy stability at the cost of increased stochasticity. Our framework provides a unified explanation for recent empirical findings on deceptive reasoning, instruction-following trade-offs, and RLHF-induced sophistry, and is further validated through perturbation experiments in multi-reward RL. This work advances policy-stability analysis from empirical heuristics towards a principled theory, offering essential insights for designing safer and more trustworthy AI systems.
Figures
Figures from the paper (6 more)
Forward citations
Cited by 1 Pith paper
-
Beyond Binary: Turning Partial Success into Dense Verifiable Rewards for Reinforcement Learning in Code Generation
VeRPO turns partial test-suite success into dense, difficulty-weighted rewards for RL code generation, beating GRPO by up to 8.8 pass@1 points at negligible cost.
Reference graph
Works this paper leans on
-
[1]
L1: Controlling how long a reasoning model thinks with reinforcement learning
Pranjal Aggarwal and Sean Welleck. L1: Controlling how long a reasoning model thinks with reinforcement learning. arXiv preprint arXiv:2503.04697,
-
[3]
Constitutional ai: Harmlessness from ai feedback
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073,
-
[5]
Defense against reward poisoning attacks in reinforcement learning
Kiarash Banihashem, Adish Singla, and Goran Radanovic. Defense against reward poisoning attacks in reinforcement learning. arXiv preprint arXiv:2102.05776,
-
[7]
Scaling reasoning, losing control: Evaluating instruction following in large reasoning models
Tingchen Fu, Jiawei Gu, Yafu Li, Xiaoye Qu, and Yu Cheng. Scaling reasoning, losing control: Evaluating instruction following in large reasoning models. arXiv preprint arXiv:2505.14810,
-
[8]
Deliberative alignment: Reasoning enables safer language models
Melody Y Guan, Manas Joglekar, Eric Wallace, Saachi Jain, Boaz Barak, Alec Helyar, Rachel Dias, Andrea Vallone, Hongyu Ren, Jason Wei, et al. Deliberative alignment: Reasoning enables safer language models. arXiv preprint arXiv:2412.16339,
-
[15]
Crossing the reward bridge: Expanding rl with verifiable rewards across diverse domains
Yi Su, Dian Yu, Linfeng Song, Juntao Li, Haitao Mi, Zhaopeng Tu, Min Zhang, and Dong Yu. Crossing the reward bridge: Expanding rl with verifiable rewards across diverse domains. arXiv preprint arXiv:2503.23829,
-
[16]
Stop overthinking: A survey on efficient reasoning for large language models
Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen Zhong, Hanjie Chen, et al. Stop overthinking: A survey on efficient reasoning for large language models. arXiv preprint arXiv:2503.16419,
-
[18]
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388,
Show all 20 references
-
[19]
31 THE LLM P OLICY CLIFF A Related Work Stability in Reinforcement Learning. A significant body of research in reinforcement learning theory focuses on the policy optimization process, analyzing convergence to optimality and the sample efficiency of various algorithms (Sutton ...
1998
-
[20]
spurious reasoning
established the practical viability of using RL to steer model behavior. This trend continues with the use of Reinforcement Learning with Verifiable Rewards (RLVR) in state-of-the-art large reasoning models, including OpenAI’s o-series, Gemini 2.5, Grok 4, and DeepSeek-R1. Des...
2025
-
[1963]
Towards reasoning era: A survey of long chain-of-thought for reasoning large language models
Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng, Jiannan Guan, Peng Wang, Mengkang Hu, Yuhang Zhou, Te Gao, and Wanxiang Che. Towards reasoning era: A survey of long chain-of-thought for reasoning large language models. arXiv preprint arXiv:2503.09567, 2025a. Yanda Chen, Joe ...
-
[2005]
A survey of efficient reasoning for large reasoning models: Language, multimodality, and beyond
Xiaoye Qu, Yafu Li, Zhaochen Su, Weigao Sun, Jianhao Yan, Dongrui Liu, Ganqu Cui, Daizong Liu, Shuxian Liang, Junxian He, et al. A survey of efficient reasoning for large reasoning models: Language, multimodality, and beyond. arXiv preprint arXiv:2503.21614,
-
[2013]
Proximal policy optimization algorithms
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,
-
[2017]
Hybridflow: A flexible and efficient rlhf framework
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19256,
-
[2018]
Openrlhf: An easy-to-use, scalable and high-performance rlhf framework
Jian Hu, Xibin Wu, Zilin Zhu, Xianyu, Weixun Wang, Dehao Zhang, and Yu Cao. Openrlhf: An easy-to-use, scalable and high-performance rlhf framework. arXiv preprint arXiv:2405.11143,
-
[2020]
Chi, Samuel Miserendino, Johannes Heidecke, Tejal Patwardhan, and Dan Mossing
30 THE LLM P OLICY CLIFF Miles Wang, Tom Dupré la Tour, Olivia Watkins, Alex Makelov, Ryan A. Chi, Samuel Miserendino, Johannes Heidecke, Tejal Patwardhan, and Dan Mossing. Persona features control emergent misalignment. OpenAI, 2025a. Yiping Wang, Qing Yang, Zhiyuan Zeng, Lil...
-
[2022]
Monitoring reasoning models for misbehavior and the risks of promoting obfuscation
Bowen Baker, Joost Huizinga, Leo Gao, Zehao Dou, Melody Y Guan, Aleksander Madry, Wojciech Zaremba, Jakub Pachocki, and David Farhi. Monitoring reasoning models for misbehavior and the risks of promoting obfuscation. arXiv preprint arXiv:2503.11926,
-
[2023]
Modomodo: Multi-domain data mixtures for multimodal llm reinforcement learning
Yiqing Liang, Jielin Qiu, Wenhao Ding, Zuxin Liu, James Tompkin, Mengdi Xu, Mengzhou Xia, Zhengzhong Tu, Laixi Shi, and Jiacheng Zhu. Modomodo: Multi-domain data mixtures for multimodal llm reinforcement learning. arXiv preprint arXiv:2505.24871,
-
[2024]
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948,
-
[2025]
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al
System card documentation. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923,
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.