pith. machine review for the scientific record. sign in

arxiv: 2605.02320 · v2 · submitted 2026-05-04 · 💻 cs.AI · cs.LG

Recognition: 3 theorem links

· Lean Theorem

ANO: A Principled Approach to Robust Policy Optimization

Yiheng Zhang , Yiming Wang , Kaiyan Zhao , Zhenglin Wan , Jiayu Chen , Leong Hou U
Authors on Pith no claims yet

Pith reviewed 2026-05-08 19:30 UTC · model grok-4.3

classification 💻 cs.AI cs.LG
keywords policy optimizationreinforcement learningrobust estimationgradient clippingPPORLHFstability
0
0 comments X

The pith

Anchored Neighborhood Optimization replaces hard clipping with redescending gradients to stabilize policy learning.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper aims to fix the core dilemma in policy optimization where hard clipping wastes useful gradient information and unconstrained updates cause instability and collapse. It establishes geometric principles requiring any robust estimator to suppress outliers while preserving a smooth restoration force toward the update neighborhood. From these principles the authors derive Anchored Neighborhood Optimization as a direct replacement for clipping. If the approach holds, training can proceed at much higher learning rates without the usual failures, improving reliability in both standard reinforcement learning and large-scale alignment tasks.

Core claim

Anchored Neighborhood Optimization (ANO) is derived from a principled design space showing that a robust estimator must suppress outliers while maintaining a smooth restoration force. ANO replaces PPO's hard clipping with a redescending gradient mechanism, achieving state-of-the-art robustness in continuous and discrete control environments while uniquely preventing policy collapse at aggressive learning rates such as 1e-3. In RLHF it eliminates the catastrophic KL divergence explosion of unconstrained methods and records higher head-to-head win rates than PPO, SPO, and GRPO.

What carries the argument

The redescending gradient mechanism inside the Anchored Neighborhood Optimization framework, which smoothly reduces the weight of extreme updates instead of abruptly discarding them.

If this is right

  • ANO prevents policy collapse even under learning rates of 1 times 10 to the minus 3 in both continuous and discrete control tasks.
  • In LLM alignment ANO removes the catastrophic KL divergence explosion that unconstrained methods produce.
  • Head-to-head comparisons show ANO outperforming PPO, SPO, and GRPO in win rates across tested domains.
  • The method establishes robust state-of-the-art performance without requiring manual tuning of clipping thresholds.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same redescending principle could be adapted to stabilize other noisy gradient settings such as large-batch supervised training.
  • Higher learning rates enabled by ANO might shorten overall training time in large-scale reinforcement learning pipelines.
  • The geometric design space offers a template for creating robust variants of other first-order optimizers.

Load-bearing premise

That a robust policy optimizer must combine outlier suppression with a smooth restoration force as required by the geometric design space.

What would settle it

Experiments at learning rates of 1e-3 or higher in MuJoCo or Atari where ANO exhibits policy collapse or KL divergence explosion comparable to baselines would disprove the claims.

Figures

Figures reproduced from arXiv: 2605.02320 by Jiayu Chen, Kaiyan Zhao, Leong Hou U, Yiheng Zhang, Yiming Wang, Zhenglin Wan.

Figure 1
Figure 1. Figure 1: Geometry of Shaping Functions. A visual comparison of PPO (Hard Clip), SPO (Quadratic), and ANO (Redescending). ANO offers a unique profile that is smooth, bounded, and redescending, effectively reclaiming information from outliers without gradient explosion. 1. C∞ Smoothness (Adhering to Principle 1). Unlike PPO, which in￾troduces non-differentiable “kinks” at 1 + ϵ, fANO is constructed from ele￾mentary e… view at source ↗
Figure 2
Figure 2. Figure 2: Aggregate Performance on Traditional RL Benchmarks (5 Seeds). We report the Mean and Interquartile Mean (IQM) of normalized scores with 95% stratified bootstrap confidence intervals. Across both domains, ANO (red) achieves state-of-the-art Mean scores while maintaining highly competitive IQM, demonstrating a superior exploration ceiling without sacrificing stability. Score (4.514 vs. SPO’s 3.783), unlockin… view at source ↗
Figure 3
Figure 3. Figure 3: Robustness Analysis on MuJoCo. ANO demonstrates exceptional robustness under high learning rates (1e-3), whereas PPO suffers catastrophic performance degradation. T=0 T=0.7 T=1.0 T=0 T=0.7 T=1.0 T=0 T=0.7 T=1.0 T=0 T=0.7 T=1.0 0% 20% 40% 60% 80% 100% Response Proportion 59.5% 60.1% 59.8% 59.2% 57.6% 56.2% 76.5% 76.8% 82.2% 49.3% 56.9% 68.0% 50% Threshold ANO vs. PPO ANO vs. SPO ANO vs. GRPO GANO vs. GRPO H… view at source ↗
Figure 5
Figure 5. Figure 5: Training Dynamics. ANO maintains stable KL divergence and structured entropy. • Mitigating Goodhart’s Law: PPO achieves higher proxy rewards (∼ 7.0) than ANO (∼ 5.2) but loses head-to-head, confirming PPO suffers from reward hacking while ANO prioritizes alignment fidelity. • Semantic Stability (KL): PPO and SPO both breach the trust region (KL > 50) but for opposite reasons: PPO suffers drift due to zero … view at source ↗
Figure 6
Figure 6. Figure 6: Head-to-Head Win Rates. The left and right panels display results for ANO with ϵ = 0.2 and ϵ = 0.3, respectively. ANO consistently outperforms all baselines across all sampling temperatures (T ∈ {0, 0.7, 1.0}) and ϵ settings, further demonstrating its robustness. 13 view at source ↗
Figure 7
Figure 7. Figure 7: Full Aggregate Performance on MuJoCo (6 Environments, 5 Seeds). We report both IQM and Mean of Expert Normalized Scores (ENS). ANO demonstrates consistent superiority over PPO, TRPO, and SPO across both metrics. 0 1 2 3 4 5 6 7 ANO_0.3 PAPO_0.001 ANO_0.2 ANO_0.1 SPO_0.2 PPO_0.1 SPO_0.1 PAPO_0.01 PAPO_0.005 PPO_0.2 PPO_0.3 SPO_0.3 TRPO_0.01 TRPO_0.02 TRPO_0.03 4.514 4.110 3.967 3.933 3.783 3.775 3.709 3.686… view at source ↗
Figure 8
Figure 8. Figure 8: Full Aggregate Performance on Atari (40 Environments, 5 Seeds). We report both IQM and Mean of Human Normalized Scores (HNS). ANO achieves the highest aggregate mean scores and great IQM scores, proving its effectiveness in high-dimensional discrete control. D Algorithm Implementation 15 view at source ↗
Figure 9
Figure 9. Figure 9: Every Performance on MuJoCo (6 Environments, 5 Seeds). Algorithm 1 Training Procedure for Anchored Neighborhood Optimization (ANO) 1: Require: Initial parameters θ0, ϕ0; clipping/shaping threshold ϵ; coefficients λval, λent 2: Hyperparameters: Learning rates αθ, αϕ; Discount γ, GAE parameter λ 3: for iteration k = 0, 1, 2, . . . do 4: # 1. Interaction & Data Collection 5: Sample trajectories T = {(st, at, … view at source ↗
Figure 10
Figure 10. Figure 10: Every Performance on Atari (40 Environments, 5 Seeds). 17 view at source ↗
read the original abstract

Proximal Policy Optimization (PPO) dominates reinforcement learning and LLM alignment but relies on a "hard clipping" mechanism that discards valuable gradients. Conversely, unconstrained methods like SPO expose the optimization to unbounded updates, causing severe instability and policy collapse during extreme outlier encounters. To resolve this dilemma, we introduce a principled design space for policy optimization, demonstrating that a robust estimator must inherently suppress outliers while maintaining a smooth restoration force. Guided by these geometric principles, we derive Anchored Neighborhood Optimization (ANO), a novel method that seamlessly replaces hard clipping with a redescending gradient mechanism. Extensive evaluations demonstrate ANO's empirical superiority across diverse domains. In continuous (MuJoCo) and discrete (Atari) control, ANO establishes a robust state-of-the-art, uniquely preventing policy collapse even under highly aggressive learning rates ($1 \times 10^{-3}$). Furthermore, in LLM alignment (RLHF), ANO explicitly eliminates the catastrophic KL divergence explosion inherent to unconstrained methods, dominating PPO, SPO, and GRPO in head-to-head win rates.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. The paper introduces Anchored Neighborhood Optimization (ANO) as a replacement for the hard clipping mechanism in Proximal Policy Optimization (PPO). It posits a design space for robust policy optimization grounded in geometric principles stating that a robust estimator must suppress outliers while maintaining a smooth restoration force. From these principles, the authors derive a redescending gradient mechanism that avoids both the gradient discarding of clipping and the instability of unconstrained methods like SPO. The manuscript claims that ANO achieves state-of-the-art performance in continuous control (MuJoCo), discrete control (Atari), and LLM alignment (RLHF), uniquely preventing policy collapse under aggressive learning rates and outperforming PPO, SPO, and GRPO in head-to-head comparisons.

Significance. If the derivation is shown to be deductive rather than suggestive and the empirical results are reproducible with proper controls, ANO could provide a more stable and principled alternative to PPO for high-variance policy optimization tasks, particularly in LLM alignment where KL divergence explosions are a practical concern. The geometric framing may also offer a template for designing other robust estimators in reinforcement learning.

major comments (3)
  1. [Derivation of ANO and guiding principles] The geometric principles (outlier suppression with smooth restoration) are stated in the abstract and introduction but lack a formal definition of the design space, the neighborhood metric, or the exact optimization objective from which the redescending influence function is uniquely derived. Without this, it is unclear whether the principles entail the specific ANO mechanism or merely motivate it post hoc (see derivation of ANO and guiding principles sections).
  2. [Experimental Evaluation] The abstract asserts empirical superiority, state-of-the-art results, and dominance in win rates across MuJoCo, Atari, and RLHF, yet the manuscript supplies no experimental details, baselines, statistical tests, ablation studies, or hyperparameter settings. This renders the central empirical claims unverifiable from the provided text.
  3. [Results and Discussion] The claim that ANO 'uniquely prevents policy collapse even under highly aggressive learning rates (1e-3)' requires explicit comparison to adaptive trust-region or smoothed-clipping alternatives that might also satisfy the stated geometric principles; the current presentation does not demonstrate uniqueness.
minor comments (2)
  1. [Method] Notation for the redescending gradient and anchored neighborhood should be defined with explicit equations rather than descriptive text to improve reproducibility.
  2. [Conclusion] The manuscript would benefit from a clear statement of limitations, including any assumptions on the policy class or reward scale that the geometric principles rely upon.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive and detailed feedback. We address each major comment below and will revise the manuscript accordingly to improve clarity and rigor.

read point-by-point responses

  1. Referee: [Derivation of ANO and guiding principles] The geometric principles (outlier suppression with smooth restoration) are stated in the abstract and introduction but lack a formal definition of the design space, the neighborhood metric, or the exact optimization objective from which the redescending influence function is uniquely derived. Without this, it is unclear whether the principles entail the specific ANO mechanism or merely motivate it post hoc (see derivation of ANO and guiding principles sections).

    Authors: We agree that a more formal derivation would strengthen the paper. In the revision, we will add an explicit section defining the design space: the neighborhood is the set of policies whose divergence from the current policy is bounded by a metric (e.g., KL or total variation), and the objective is to minimize a robust surrogate loss whose influence function satisfies the geometric conditions. We will derive the redescending gradient step-by-step from the requirements that the influence function ψ(·) → 0 for large |·| (outlier suppression) while remaining smooth and positive near zero (restoration force), showing that this entails the specific ANO form under the anchored-neighborhood assumption rather than merely motivating it post hoc. revision: yes

  2. Referee: [Experimental Evaluation] The abstract asserts empirical superiority, state-of-the-art results, and dominance in win rates across MuJoCo, Atari, and RLHF, yet the manuscript supplies no experimental details, baselines, statistical tests, ablation studies, or hyperparameter settings. This renders the central empirical claims unverifiable from the provided text.

    Authors: The referee is correct that the current manuscript version does not supply sufficient experimental details in the main text. We will revise by expanding the experimental section to include all baselines (PPO, SPO, GRPO), full hyperparameter tables, statistical tests (means, standard errors, and significance over multiple random seeds), ablation studies on the redescending parameter, and reproducibility instructions. These will be summarized in the main body with explicit references to the appendix. revision: yes

  3. Referee: [Results and Discussion] The claim that ANO 'uniquely prevents policy collapse even under highly aggressive learning rates (1e-3)' requires explicit comparison to adaptive trust-region or smoothed-clipping alternatives that might also satisfy the stated geometric principles; the current presentation does not demonstrate uniqueness.

    Authors: We acknowledge that the uniqueness claim would be more convincing with direct comparisons. While the geometric principles (simultaneous redescending suppression and smooth restoration) are not satisfied by standard adaptive trust-region methods (which lack redescending behavior for extreme outliers) or simple smoothed clipping (which may not fully restore gradients), we will add an expanded discussion section explicitly contrasting ANO with these alternatives and explaining the distinctions. We will also include additional comparative runs where computationally feasible. revision: partial

Circularity Check

0 steps flagged

No circularity: derivation from stated geometric principles is independent

full rationale

The paper introduces a design space and geometric principles (robust estimator suppresses outliers while maintaining smooth restoration force), then derives ANO as a redescending mechanism replacing hard clipping. No quoted equations or steps reduce the claimed result to its inputs by construction, no fitted parameters are relabeled as predictions, and no load-bearing self-citations or uniqueness theorems from prior author work are invoked. The derivation chain remains self-contained with external empirical benchmarks on MuJoCo, Atari, and RLHF tasks.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

Abstract provides no explicit free parameters, axioms beyond the guiding geometric principle, or invented entities; the method itself is the primary contribution.

axioms (1)
  • domain assumption A robust estimator must inherently suppress outliers while maintaining a smooth restoration force
    Invoked as the foundation for the principled design space leading to ANO.

pith-pipeline@v0.9.0 · 5496 in / 1108 out tokens · 64669 ms · 2026-05-08T19:30:32.267610+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

46 extracted references · 10 canonical work pages · 6 internal anchors

  1. [1]

    Deep reinforcement learning at the edge of the statistical precipice.Advances in neural information processing systems, 34:29304–29320, 2021

    Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Belle- mare. Deep reinforcement learning at the edge of the statistical precipice.Advances in neural information processing systems, 34:29304–29320, 2021

    2021

  2. [2]

    The arcade learning environment: An evaluation platform for general agents.Journal of artificial intelligence research, 47:253–279, 2013

    Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents.Journal of artificial intelligence research, 47:253–279, 2013

    2013

  3. [3]

    Pythia: A suite for analyzing large language models across training and scaling

    Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pages 2397–2430. PMLR, 2023

    2023

  4. [4]

    Training Diffusion Models with Reinforcement Learning

    Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning.arXiv preprint arXiv:2305.13301, 2023

    work page internal anchor Pith review arXiv 2023

  5. [5]

    Optimization methods for large-scale machine learning.SIAM review, 60(2):223–311, 2018

    Léon Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning.SIAM review, 60(2):223–311, 2018

    2018

  6. [6]

    Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine

    Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep policy gradients: A case study on ppo and trpo.arXiv preprint arXiv:2005.12729, 2020

    work page arXiv 2005

  7. [7]

    Addressing function approximation error in actor-critic methods

    Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Jennifer Dy and Andreas Krause, editors,Proceedings of the 35th International Conference on Machine Learning, volume 80 ofProceedings of Machine Learning Research, pages 1587–1596. PMLR, 10–15 Jul 2018. URL https://proceedings. mlr.press/v8...

    2018

  8. [8]

    Accelerate: Training and inference at scale made simple, efficient and adaptable.https://github.com/huggingface/accelerate, 2022

    Sylvain Gugger, Lysandre Debut, Thomas Wolf, Philipp Schmid, Zachary Mueller, Sourab Mangrulkar, Marc Sun, and Benjamin Bossan. Accelerate: Training and inference at scale made simple, efficient and adaptable.https://github.com/huggingface/accelerate, 2022

    2022

  9. [9]

    Jonathan Ho and Stefano Ermon

    Nicolas Heess, Dhruva Tb, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, SM Eslami, et al. Emergence of locomotion behaviours in rich environments.arXiv preprint arXiv:1707.02286, 2017

    work page arXiv 2017

  10. [10]

    Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms.Journal of Machine Learning Research, 23(274):1–18, 2022

    Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and JoÃG, o GM AraÚjo. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms.Journal of Machine Learning Research, 23(274):1–18, 2022

    2022

  11. [11]

    Robust estimation of a location parameter

    Peter J Huber. Robust estimation of a location parameter. InBreakthroughs in statistics: Methodology and distribution, pages 492–518. Springer, 1992

    1992

  12. [12]

    Robust statistics

    Peter J Huber. Robust statistics. InInternational encyclopedia of statistical science, pages 1248–1251. Springer, 2011

    2011

  13. [13]

    Are Deep Policy Gradient Algorithms Truly Policy Gradient Algorithms? , Year =

    Andrew Ilyas, Logan Engstrom, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Are deep policy gradient algorithms truly policy gradient algorithms.arXiv preprint arXiv:1811.02553, 2018

    work page arXiv 2018

  14. [14]

    Approximately optimal approximate reinforcement learning

    Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. InProceedings of the nineteenth international conference on machine learning, pages 267–274, 2002

    2002

  15. [15]

    Hyper- spherical normalization for scalable deep reinforcement learning

    Hojoon Lee, Youngdo Lee, Takuma Seno, Donghu Kim, Peter Stone, and Jaegul Choo. Hyper- spherical normalization for scalable deep reinforcement learning. 2025

    2025

  16. [16]

    DeepSeek-V3 Technical Report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024. 10

    work page Pith review arXiv 2024

  17. [17]

    Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning

    Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al. Isaac gym: High performance gpu-based physics simulation for robot learning.arXiv preprint arXiv:2108.10470, 2021

    work page internal anchor Pith review arXiv 2021

  18. [18]

    Human-level control through deep reinforcement learning.nature, 518(7540):529–533, 2015

    V olodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning.nature, 518(7540):529–533, 2015

    2015

  19. [19]

    Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730–27744, 2022

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730–27744, 2022

    2022

  20. [20]

    On the difficulty of training recurrent neural networks

    Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. InInternational conference on machine learning, pages 1310–1318. Pmlr, 2013

    2013

  21. [21]

    Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters

    Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 3505–3506, 2020

    2020

  22. [22]

    Learning to walk in minutes using massively parallel deep reinforcement learning

    Nikita Rudin, David Hoeller, Philipp Reist, and Marco Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning. InConference on robot learning, pages 91–100. PMLR, 2022

    2022

  23. [23]

    Trust region policy optimization

    John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. InInternational conference on machine learning, pages 1889–1897. PMLR, 2015

    2015

  24. [24]

    High-Dimensional Continuous Control Using Generalized Advantage Estimation

    John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation.arXiv preprint arXiv:1506.02438, 2015

    work page internal anchor Pith review arXiv 2015

  25. [25]

    Proximal Policy Optimization Algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    work page internal anchor Pith review arXiv 2017

  26. [26]

    DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    work page internal anchor Pith review arXiv 2024

  27. [27]

    Mas- tering the game of go with deep neural networks and tree search.nature, 529(7587):484–489, 2016

    David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mas- tering the game of go with deep neural networks and tree search.nature, 529(7587):484–489, 2016

    2016

  28. [28]

    A general reinforcement learning algorithm that masters chess, shogi, and go through self-play.Science, 362(6419):1140–1144, 2018

    David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play.Science, 362(6419):1140–1144, 2018

    2018

  29. [29]

    Learning to summarize with human feedback

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea V oss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in neural information processing systems, 33:3008–3021, 2020

    2020

  30. [30]

    Mujoco: A physics engine for model-based control

    Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In2012 IEEE/RSJ international conference on intelligent robots and systems, pages 5026–5033. IEEE, 2012

    2012

  31. [31]

    Gymnasium: A Standard Interface for Reinforcement Learning Environments

    Mark Towers, Ariel Kwiatkowski, Jordan Terry, John U Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Markus Krimmel, Arjun KG, et al. Gymnasium: A standard interface for reinforcement learning environments.arXiv preprint arXiv:2407.17032, 2024. 11

    work page internal anchor Pith review arXiv 2024

  32. [32]

    Grandmaster level in starcraft ii using multi-agent reinforcement learning.nature, 575(7782):350–354, 2019

    Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Jun- young Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning.nature, 575(7782):350–354, 2019

    2019

  33. [33]

    TRL: Transformers Rein- forcement Learning, 2020

    Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. TRL: Transformers Rein- forcement Learning, 2020. URLhttps://github.com/huggingface/trl

    2020

  34. [34]

    Truly proximal policy optimization

    Yuhui Wang, Hao He, and Xiaoyang Tan. Truly proximal policy optimization. InUncertainty in artificial intelligence, pages 113–122. PMLR, 2020

    2020

  35. [35]

    Tianshou: A highly modularized deep reinforcement learning library

    Jiayi Weng, Huayu Chen, Dong Yan, Kaichao You, Alexis Duburcq, Minghao Zhang, Yi Su, Hang Su, and Jun Zhu. Tianshou: A highly modularized deep reinforcement learning library. Journal of Machine Learning Research, 23(267):1–6, 2022. URLhttp://jmlr.org/papers/ v23/21-1127.html

    2022

  36. [36]

    Envpool: A highly parallel reinforcement learning environment execution engine.Advances in Neural Information Processing Systems, 35:22409–22421, 2022

    Jiayi Weng, Min Lin, Shengyi Huang, Bo Liu, Denys Makoviichuk, Viktor Makoviychuk, Zichen Liu, Yufan Song, Ting Luo, Yukun Jiang, et al. Envpool: A highly parallel reinforcement learning environment execution engine.Advances in Neural Information Processing Systems, 35:22409–22421, 2022

    2022

  37. [37]

    Simple policy optimization

    Zhengpeng Xie, Qiang Zhang, Fan Yang, Marco Hutter, and Renjing Xu. Simple policy optimization. InForty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=SG8Yx1FyeU

    2025

  38. [38]

    Mastering complex control in moba games with deep reinforcement learning

    Deheng Ye, Zhao Liu, Mingfei Sun, Bei Shi, Peilin Zhao, Hao Wu, Hongsheng Yu, Shaojie Yang, Xipeng Wu, Qingwei Guo, et al. Mastering complex control in moba games with deep reinforcement learning. InProceedings of the AAAI conference on artificial intelligence, volume 34, pages 6672–6679, 2020

    2020

  39. [39]

    Why gradient clipping accelerates training: A theoretical justification for adaptivity

    Jingzhao Zhang, Tianxing He, Suvrit Sra, and Ali Jadbabaie. Why gradient clipping accelerates training: A theoretical justification for adaptivity. InInternational Conference on Learning Representations

  40. [40]

    Absolute policy optimization: Enhancing lower probability bound of performance with high confidence

    Weiye Zhao, Feihan Li, Yifan Sun, Rui Chen, Tianhao Wei, and Changliu Liu. Absolute policy optimization: Enhancing lower probability bound of performance with high confidence. In Forty-first International Conference on Machine Learning, 2024. 12 A Experimental Details In this section, we provide the comprehensive implementation details, hyperparameter set...

    2024

  41. [41]

    X s X a ρπ(s)pπ(a|s) min g(r)Aπ(s, a), f(r)A π(s, a) − X s X a ρπ(s)pπ(a|s) δ[1−ϵl,1+ϵu](r) # = argmax ˜π

    and random scores are from Lee et al. [15]. For Atari, we apply Human Normalized Score (HNS): HNS= Agent Score−Random Score Human Score−Random Score ,(13) where the human scores and random scores are from Mnih et al. [18]. To ensure a strictly fair comparison between ANO, PPO, and GRPO, we aligned theGlobal Batch SizeandTotal Training Episodesacross all a...

  42. [42]

    Thus, the derivation is exact

    Fora 3 (Lower Bound:˜π3 =π 3(1−ϵ)): −6 + 4(1−α) 0.1(1−0.6) + 2 = 0 =⇒ −4 + 4(1−α) 0.04 = 0 =⇒1−α= 0.04 =⇒α= 0.96 Both conditions consistently yieldα= 0.96. Thus, the derivation is exact. H Proofs and Derivations for ANO Recall the definition of the base kernel: ϕ(z) := ln(1 + 2 −2z) + 4 1 + 2−z .(27) The shaping functionf ANO(r)is defined as: fANO(r) = 45...

  43. [43]

    By the Intermediate Value Theorem, there exists at least one rootx ∗ ∈(0,1)

    Existence: P(0) =−1<0 and P(1) = 8>0 . By the Intermediate Value Theorem, there exists at least one rootx ∗ ∈(0,1)

  44. [44]

    Thus, P(x) is strictly monotonically increasing on positive reals, implying the root x∗ is unique

    Uniqueness:The derivative P ′(x) = 5x 4 + 15x2 + 2x+ 2 is strictly positive for all x >0 . Thus, P(x) is strictly monotonically increasing on positive reals, implying the root x∗ is unique. Since the mapping r↔x is a bijection, the unique solution x∗ corresponds to a unique state ratio r∗. Thus,f ANO(r)changes its convexity exactly once. H.4 Proof of Boun...

  45. [45]

    Bounded Maximization (Principle 2):The set of maximizers is bounded above by r∗, and forr > r ∗,f(r)is strictly decreasing

  46. [46]

    Then, f cannot be globally concave on the tail interval (r∗,+∞)

    Asymptotic Stability (Principle 3 & A):The altitude of (sub)gradient decays to 0 as r→ +∞. Then, f cannot be globally concave on the tail interval (r∗,+∞) . It must exhibitat least onechange in convexity (inflection point) in this region. Proof. We proceed by contradiction. Assume that f(r) is globally concave on the interval (r∗,+∞) . Since f is strictly...