Pith. sign in

REVIEW 1 major objections 5 minor 46 references

Stop-gradient self-anchor breaks RL clipping ceiling for LLM reasoning

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

2026-07-09 22:06 UTC pith:37BEJ4IN

load-bearing objection UP replaces the IS ratio with a stop-gradient self-anchor for positive advantages, recovering an unclipped REINFORCE-style gradient — but the 'equivalence to REINFORCE' claim papers over a sampling distribution mismatch that introduces off-policy bias. the 1 major comments →

arxiv 2607.06987 v1 pith:37BEJ4IN submitted 2026-07-08 cs.LG

UP: Unbounded Positive Asymmetric Optimization for Breaking the Exploration-Stability Dilemma

Chongyu Fan , Pengfei Liu , Jingjia Huang , Sijia Liu , Yi Lin This is my paper
classification cs.LG
keywords reinforcement learningimportance samplingpolicy gradientstop-gradient operatorexploration-stability dilemmaasymmetric optimizationlarge language modelsreasoning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Modern reinforcement learning for large language models faces a structural tension: importance sampling enables reuse of sampled data but introduces gradient instability on rare reasoning paths, while clipping mechanisms that prevent this instability also cap how much the model can reinforce correct but low-confidence trajectories. This paper formalizes that cap as Probability Capacity, the maximum allowable probability change before the gradient is truncated, and shows it is linearly bottlenecked by the historical policy. The authors propose replacing the standard importance sampling ratio (current policy divided by historical policy) with a self-anchored ratio (current policy divided by a stop-gradient copy of itself) for positive-advantage rollouts. The stop-gradient operator makes the denominator act as a constant during backpropagation, so the ratio's gradient reduces exactly to the stable, unclipped REINFORCE gradient. This removes the clipping ceiling for correct rollouts while retaining standard clipping for incorrect ones, creating an asymmetric objective that maximizes exploration without sacrificing stability.

Core claim

The central mechanism is the identity that when the importance sampling ratio is rewritten as π_θ / sg(π_θ), where sg is the stop-gradient operator, its gradient with respect to θ becomes identical to the gradient of log π_θ. This means the unclipped self-anchored ratio is mathematically equivalent to on-policy REINFORCE, but it can still be computed using data sampled from a historical policy. The paper proves this equivalence at both token-level and sequence-level granularity, and shows that the resulting Probability Capacity becomes 1 − π_θ (unconstrained by the historical policy) rather than the clipping-limited (1 + ε)π_old. The asymmetric design applies this unbounded update only to正确回

What carries the argument

The stop-gradient operator sg(·), which treats its argument as a constant during backpropagation, converting the ratio π_θ/sg(π_θ) into a form whose gradient equals ∇log π_θ. Combined with asymmetric routing: unbounded positive branch for correct rollouts, standard clipped branch for incorrect rollouts.

Load-bearing premise

The method assumes that unbounded updates for positive advantages are always safe as long as negative advantages remain clipped, but this is only tested on mathematical reasoning tasks with well-defined correctness signals. Whether spurious or adversarial high-reward paths could exploit the unbounded positive channel to destabilize training is not examined.

What would settle it

If a training scenario exists where a positive-advantage rollout is actually incorrect (mislabeled or reward-hacked), the unbounded gradient would aggressively reinforce a wrong trajectory with no clipping safeguard, potentially causing representation collapse that the negative-advantage clipping cannot prevent.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the self-anchored ratio preserves stability while removing clipping, it could replace the standard IS ratio in any policy gradient method that uses clipping, not just GRPO/DAPO/GSPO.
  • The Probability Capacity formalism provides a diagnostic tool: any clipping-based RL algorithm can be analyzed for how much exploration budget it structurally denies to low-confidence correct paths.
  • The asymmetric principle (unbounded for positive, clipped for negative) suggests that the two directions of policy update have fundamentally different stability requirements, challenging the symmetric design assumption in most RL objectives.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If reward hacking produces spurious high-advantage rollouts, the unbounded positive branch would amplify them without any clipping brake, potentially making the method more vulnerable to reward misspecification than symmetric clipping algorithms.
  • The self-anchoring trick might generalize beyond importance sampling ratios: any ratio of the form f(θ)/g(θ) where g is meant to be a reference could potentially be replaced by f(θ)/sg(f(θ)) to recover a stable gradient, though whether this is desirable depends on the specific application.
  • The fact that entropy remains high throughout training suggests the method may avoid the mode collapse that limits test-time compute scaling, which could matter for inference-time reasoning strategies that rely on diverse sampling.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 5 minor

Summary. The paper proposes Unbounded Positive Asymmetric Optimization (UP), a plug-and-play objective for RL-based LLM training. The core idea is to replace the importance sampling ratio r = π_θ/π_old with a self-anchored ratio r̃ = π_θ/sg(π_θ) for positive advantages, where sg(·) is the stop-gradient operator. The authors show that this yields a gradient equivalent to REINFORCE (Eq. 13), eliminating the variance explosion of unclipped IS while removing the exploration-stifling upper clip. For negative advantages, standard clipping is retained as a safeguard. The method is instantiated for DAPO, GRPO, and GSPO, and evaluated on math and geometric reasoning across dense, MoE, and vision-language models. The Probability Capacity (Cap) formalization (Eq. 11) is introduced to quantify how clipping constrains the update budget for low-probability tokens. Experiments show consistent gains over baselines with comparable or lower gradient norms and KL divergence.

Significance. The paper addresses a well-known tension in IS-based RL for LLMs: clipping prevents instability but stifles exploration of rare correct trajectories. The stop-gradient self-anchoring trick is a clean, parameter-free, and easily implementable solution. The derivation in Eq. 13 is correct as a calculus identity, and the Probability Capacity analysis provides a useful lens. The experimental coverage is broad: three algorithms, three architectures, two modalities, and eleven baselines. The ablations in §5.3 isolating the self-anchored ratio and the asymmetric design are well-designed and directly support the central claims. The main theoretical concern—off-policy bias from sampling under π_old while computing a REINFORCE-form gradient—is real but does not negate the method's empirical effectiveness; it does require reframing the theoretical contribution.

major comments (1)
  1. §4.1, Eq. 12–13: The claim that the UP gradient is 'mathematically equivalent to maximizing the REINFORCE objective established in Eq. 1' is imprecise. The gradient form matches (∇log π_θ), but the expectation is over o ∼ π_old (off-policy), whereas REINFORCE samples o ∼ π_θ (on-policy). The IS ratio r = π_θ/π_old exists precisely to correct this distributional mismatch; by replacing π_old with sg(π_θ), UP removes the IS correction, yielding a biased estimator of the on-policy policy gradient whenever π_θ ≠ π_old. This bias grows with each mini-batch step within a rollout group (Table A1 shows 16–32 mini-batch updates per rollout batch). The paper should explicitly acknowledge this off-policy bias and reframe the contribution: UP trades IS variance for off-policy bias, and the empirical success suggests this bias acts as implicit regularization rather than that UP 'completely eradicates'
minor comments (5)
  1. §3.3, Eq. 11: The Cap formula uses π_old as a scalar, but π_old is a per-token probability. Clarify that the analysis is per-token.
  2. Table 1: OlympiadBench score for UP-GRPO (58.33) is described as 'second only to ASPO's 58.48,' but the table shows ASPO at 58.48. This is correct but the phrasing could be clearer.
  3. Fig. 5: The y-axis scale for gradient norm (10^1 to 10^13) makes the stable methods hard to compare. Consider a log scale or inset.
  4. Appendix B: The Dual Clip threshold c is introduced but its value is not specified in the main text or appendix.
  5. The term 'Probability Capacity' is introduced in §3.3 but the abbreviation 'Cap' is used inconsistently; standardize.

Circularity Check

0 steps flagged

No circularity: the key derivation (Eq. 13) follows from standard calculus and the stop-gradient definition, not from fitted parameters or self-cited results.

full rationale

The paper's central theoretical claim is that replacing the IS ratio r_{i,t}(θ) = π_θ/π_old with the self-anchored ratio r̃_{i,t}(θ) = π_θ/sg(π_θ) yields a gradient equivalent to REINFORCE (Eq. 13). This derivation is self-contained: the stop-gradient operator sg(·) is defined to treat its operand as a constant during backpropagation, so ∇[π_θ/sg(π_θ)] = (1/sg(π_θ))∇π_θ = (1/π_θ)∇π_θ = ∇log π_θ by the standard log-derivative trick. No fitted parameters, no self-cited uniqueness theorems, and no ansatz smuggled through citation are needed. The Probability Capacity formalization (Eq. 11) is derived directly from the clipping definitions in GRPO/DAPO. The experimental results are measured against external benchmarks (AIME24, AMC23, MATH500, etc.) with no fitted-input-as-prediction pattern. The skeptic's concern about off-policy bias (sampling from π_old while the gradient form matches on-policy REINFORCE) is a correctness issue about whether the theoretical claim is accurately stated, not a circularity issue — the derivation itself does not reduce to its inputs by construction. Self-citations present in the paper (e.g., Ref [6], [7]) are contextual and not load-bearing for the mathematical derivation. The derivation chain is independent and non-circular.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 1 invented entities

The paper introduces one new conceptual entity (Probability Capacity) and one ad hoc design choice (asymmetric unbounded/clipped routing). The free parameters are all inherited from baseline algorithms. The mathematical axioms are standard. The main methodological axiom — that positive advantages should be unbounded while negative should be clipped — is an engineering decision validated empirically but not derived from theory.

free parameters (3)
  • ε_low (lower clip bound) = 0.2 (GRPO/UP-GRPO), 0.2 (DAPO/UP-DAPO), 3e-4 (GSPO/UP-GSPO)
    Standard PPO-family hyperparameter; not introduced by this paper but required for the negative-advantage branch.
  • ε_high (upper clip bound) = 0.28 (DAPO), 0.2 (GRPO), 4e-4 (GSPO)
    Used only for negative advantages in UP variants; eliminated for positive advantages. Standard from baseline algorithms.
  • β (KL penalty coefficient) = Not explicitly stated; inherited from GRPO baseline
    Used in UP-GRPO negative branch; standard GRPO hyperparameter.
axioms (4)
  • standard math The stop-gradient operator sg(x) returns x in forward pass but has zero gradient during backpropagation.
    Standard deep learning primitive; used in Eq. 12 to formulate the self-anchored ratio.
  • standard math Importance sampling provides an unbiased estimator of expectations under the target distribution when using samples from the proposal distribution.
    Standard statistical technique; invoked in §3.1 to justify multi-step optimization.
  • domain assumption REINFORCE gradients are mathematically stable for on-policy optimization.
    Invoked in §3.1 and §4.1 to justify the desirability of recovering the REINFORCE gradient. This is standard but assumes the policy network is well-conditioned.
  • ad hoc to paper Correct rollouts (positive advantages) should receive unbounded reinforcement while wrong rollouts (negative advantages) require clipping safeguards.
    This asymmetric design choice is the core methodological assumption. It is motivated by the analysis but not derived from first principles — it is an engineering decision supported by ablation (§5.3).
invented entities (1)
  • Probability Capacity (Cap) no independent evidence
    purpose: Formalizes the maximum allowable probability change before gradient truncation
    A conceptual tool introduced by the paper to quantify the exploration bottleneck. It is derived from existing clipping definitions (Eq. 11) and does not require independent empirical validation, but it is a new formalization.

pith-pipeline@v1.1.0-glm · 21672 in / 2523 out tokens · 335453 ms · 2026-07-09T22:06:46.696469+00:00 · methodology

0 comments
read the original abstract

Reinforcement learning (RL) has become the standard paradigm for enhancing the complex reasoning capabilities of large language models (LLMs). To achieve sample efficiency, modern RL frameworks rely on importance sampling (IS). However, these algorithms suffer from an exploration-stability dilemma. Pure IS often leads to catastrophic training instability, while standard clipping mechanisms used to mitigate this instability strictly constrain the policy update budget. By formalizing the concept of Probability Capacity (Cap), we reveal that conservative clipping structurally stifles exploration by prematurely truncating the update budget for correct but low-confidence reasoning paths. To break free from these constraints, we propose Unbounded Positive Asymmetric Optimization (UP), a universal and plug-and-play objective. UP theoretically restructures the optimization process by anchoring the policy to its current state via the stop-gradient operator. This asymmetric design unleashes unclipped, stable gradients for positive advantages to maximize exploration, while maintaining standard clipping safeguards for negative advantages to prevent training instability. Furthermore, our formulation readily extends across different optimization granularities, including token-level (GRPO, DAPO) and sequence-level (GSPO) frameworks. Extensive experiments demonstrate that UP enhances exploration capacity and achieves superior reasoning accuracy across diverse RL algorithms (DAPO, GSPO, and GRPO), model architectures (Dense, MoE, and vision-language), and training modalities (language and multimodal), validating UP as a truly universal plug-and-play enhancement for RL-based training.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

46 extracted references · 46 canonical work pages · 19 internal anchors

  1. [1]

    Back to basics: Revisiting reinforce-style optimization for learning from human feedback in llms

    Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce-style optimization for learning from human feedback in llms. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12248–12267, 2024

    work page 2024

  2. [2]

    Constitutional AI: Harmlessness from AI Feedback

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  3. [3]

    Efficient reinforcement learning with semantic and token entropy for llm reasoning.arXiv preprint arXiv:2512.04359, 2025

    Hongye Cao, Zhixin Bai, Ziyue Peng, Boyan Wang, Tianpei Yang, Jing Huo, Yuyao Zhang, and Yang Gao. Efficient reinforcement learning with semantic and token entropy for llm reasoning.arXiv preprint arXiv:2512.04359, 2025

    work page arXiv 2025

  4. [4]

    MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention

    Aili Chen, Aonian Li, Bangwei Gong, Binyang Jiang, Bo Fei, Bo Yang, Boji Shan, Changqing Yu, Chao Wang, Cheng Zhu, et al. Minimax-m1: Scaling test-time compute efficiently with lightning attention.arXiv preprint arXiv:2506.13585, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  5. [5]

    American invitational mathematics examination-aime 2024, 2024

    MAA Codeforces. American invitational mathematics examination-aime 2024, 2024

    work page 2024

  6. [6]

    Rethinking Muon Beyond Pretraining: Spectral Failures and High-Pass Remedies for VLA and RLVR

    Chongyu Fan, Gaowen Liu, Mingyi Hong, Ramana Rao Kompella, and Sijia Liu. Rethinking muon beyond pretraining: Spectral failures and high-pass remedies for vla and rlvr.arXiv preprint arXiv:2605.19282, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  7. [7]

    Hero, and Sijia Liu

    Chongyu Fan, Yihua Zhang, Jinghan Jia, Alfred O. Hero, and Sijia Liu. Cyclicreflex: Improving reasoning models via cyclical reflection token scheduling. InThe FourteenthInternational Conference on Learning Representations,

    work page

  8. [8]

    URLhttps://openreview.net/forum?id=4oOF4J2xSy

    work page

  9. [9]

    Soft Adaptive Policy Optimization

    Chang Gao, Chujie Zheng, Xiong-Hui Chen, Kai Dang, Shixuan Liu, Bowen Yu, An Yang, Shuai Bai, Jingren Zhou, and Junyang Lin. Soft adaptive policy optimization.arXiv preprint arXiv:2511.20347, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  10. [10]

    DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  11. [11]

    Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume1: Long Papers...

    work page 2024

  12. [12]

    REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization

    Jian Hu, Jason Klein Liu, Haotian Xu, and Wei Shen. Reinforce++: Stabilizing critic-free policy optimization with global advantage normalization.arXiv preprint arXiv:2501.03262, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  13. [13]

    Efficient memory management for large language model serving with pagedattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles, pages 611–626, 2023

    work page 2023

  14. [14]

    Solving quantitative reasoning problems with language models

    Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advancesin neural information processing systems, 35:3843–3857, 2022

    work page 2022

  15. [15]

    Back to basics: Revisiting exploration in reinforcement learning for llm reasoning via generative probabilities.arXiv preprint arXiv:2602.05281, 2026

    Pengyi Li, Elizaveta Goncharova, Andrey Kuznetsov, and Ivan Oseledets. Back to basics: Revisiting exploration in reinforcement learning for llm reasoning via generative probabilities.arXiv preprint arXiv:2602.05281, 2026

    work page arXiv 2026

  16. [16]

    Bandpo: Bridging trust regions and ratio clipping via probability-aware bounds for llm reinforcement learning.arXiv preprint arXiv:2603.04918, 2026

    Yuan Li, Bo Wang, Yufei Gao, Yuqian Yao, Xinyuan Wang, Zhangyue Yin, and Xipeng Qiu. Bandpo: Bridging trust regions and ratio clipping via probability-aware bounds for llm reinforcement learning.arXiv preprint arXiv:2603.04918, 2026

    work page arXiv 2026

  17. [17]

    Let’s verify step by step

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. InThe twelfth international conference on learning representations, 2023

    work page 2023

  18. [18]

    Length-unbiased sequence policy optimization: Revealing and controlling response length variation in rlvr.arXiv preprint arXiv:2602.05261, 2026

    Fanfan Liu, Youyang Yin, Peng Shi, Siqi Yang, Zhixiong Zeng, and Haibo Qiu. Length-unbiased sequence policy optimization: Revealing and controlling response length variation in rlvr.arXiv preprint arXiv:2602.05261, 2026

    work page arXiv 2026

  19. [19]

    Understanding R1-Zero-Like Training: A Critical Perspective

    Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective.arXiv preprint arXiv:2503.20783, 2025. 13

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  20. [20]

    Inter-GPS: Interpretable geometry problem solving with formal language and symbolic reasoning

    Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chun Zhu. Inter-GPS: Interpretable geometry problem solving with formal language and symbolic reasoning. InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, page...

    work page 2021

  21. [21]

    Stabilizing moe reinforcement learning by aligning training and inference routers.arXiv preprint arXiv:2510.11370, 2025

    Wenhan Ma, Hailin Zhang, Liang Zhao, Yifan Song, Yudong Wang, Zhifang Sui, and Fuli Luo. Stabilizing moe reinforcement learning by aligning training and inference routers.arXiv preprint arXiv:2510.11370, 2025

    work page arXiv 2025

  22. [22]

    Clip your sequences fairly: Enforcing length fairness for sequence-level rl.arXiv preprint arXiv:2509.09177, 2025

    Hanyi Mao, Quanjia Xiao, Lei Pang, and Haixiao Liu. Clip your sequences fairly: Enforcing length fairness for sequence-level rl.arXiv preprint arXiv:2509.09177, 2025

    work page arXiv 2025

  23. [23]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advancesin neural information processing systems, 35:27730–27744, 2022

    work page 2022

  24. [24]

    Rethinking the Trust Region in LLM Reinforcement Learning

    Penghui Qi, Xiangxin Zhou, Zichen Liu, Tianyu Pang, Chao Du, Min Lin, and Wee Sun Lee. Rethinking the trust region in llm reinforcement learning.arXiv preprint arXiv:2602.04879, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  25. [25]

    Tapered Off-Policy REINFORCE: Stable and efficient reinforcement learning for LLMs

    Nicolas Le Roux, Marc G Bellemare, Jonathan Lebensold, Arnaud Bergeron, Joshua Greaves, Alex Fréchette, Carolyne Pelletier, Eric Thibodeau-Laufer, Sándor Toth, and Sam Work. Tapered off-policy reinforce: Stable and efficient reinforcement learning for llms.arXiv preprint arXiv:2503.14286, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  26. [26]

    Trustregionpolicyoptimization

    JohnSchulman, SergeyLevine, PieterAbbeel, MichaelJordan, andPhilippMoritz. Trustregionpolicyoptimization. In International conference on machine learning, pages 1889–1897. PMLR, 2015

    work page 2015

  27. [27]

    Proximal Policy Optimization Algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  28. [28]

    Bytedance Seed. Seed2. 0 model card: Towards intelligence frontier for real-world complexity.arXiv preprint arXiv:2607.00248, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  29. [29]

    DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  30. [30]

    Hybridflow: A flexible and efficient rlhf framework

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. InProceedings of the TwentiethEuropean Conference on Computer Systems, pages 1279–1297, 2025

    work page 2025

  31. [31]

    Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  32. [32]

    Klear-reasoner: Advancing reasoning capability via gradient-preserving clipping policy optimization

    Zhenpeng Su, Leiyu Pan, Xue Bai, Dening Liu, Guanting Dong, Jiaming Huang, Wenping Hu, Fuzheng Zhang, Kun Gai, and Guorui Zhou. Klear-reasoner: Advancing reasoning capability via gradient-preserving clipping policy optimization. arXiv preprint arXiv:2508.07629, 2025

    work page arXiv 2025

  33. [33]

    When Importance Sampling Misallocates Credit: Asymmetric Ratios for Outcome-Supervised RL

    Jiakang Wang, Runze Liu, Lei Lin, Wenping Hu, Xiu Li, Fuzheng Zhang, Guorui Zhou, and Kun Gai. Aspo: Asymmetric importance sampling policy optimization.arXiv preprint arXiv:2510.06062, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  34. [34]

    Simple statistical gradient-following algorithms for connectionist reinforcement learning

    Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3):229–256, 1992

    work page 1992

  35. [35]

    An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12122, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  36. [36]

    Qwen3 Technical Report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  37. [37]

    DAPO: An Open-Source LLM Reinforcement Learning System at Scale

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  38. [38]

    Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?

    Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?arXiv preprint arXiv:2504.13837, 2025. 14

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  39. [39]

    Star: Bootstrapping reasoning with reasoning

    Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advancesin Neural Information Processing Systems, 35:15476–15488, 2022

    work page 2022

  40. [40]

    On the design of kl-regularized policy gradient algorithms for llm reasoning.arXiv preprint arXiv:2505.17508, 2025

    Yifan Zhang, Yifeng Liu, Huizhuo Yuan, Yang Yuan, Quanquan Gu, and Andrew Chi-Chih Yao. On the design of kl-regularized policy gradient algorithms for llm reasoning.arXiv preprint arXiv:2505.17508, 2025

    work page arXiv 2025

  41. [41]

    Geometric-mean policy optimization.arXiv preprint arXiv:2507.20673, 2025

    Yuzhong Zhao, Yue Liu, Junpeng Liu, Jingye Chen, Xun Wu, Yaru Hao, Tengchao Lv, Shaohan Huang, Lei Cui, Qixiang Ye, et al. Geometric-mean policy optimization.arXiv preprint arXiv:2507.20673, 2025

    work page arXiv 2025

  42. [42]

    Group Sequence Policy Optimization

    Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization.arXiv preprint arXiv:2507.18071, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  43. [43]

    Prosperity before collapse: How far can off-policy rl reach with stale data on llms?arXiv preprint arXiv:2510.01161, 2025

    Haizhong Zheng, Jiawei Zhao, and Beidi Chen. Prosperity before collapse: How far can off-policy rl reach with stale data on llms?arXiv preprint arXiv:2510.01161, 2025

    work page arXiv 2025

  44. [44]

    The surprising effectiveness of negative reinforcement in llm reasoning.arXiv preprint arXiv:2506.01347, 2025

    Xinyu Zhu, Mengzhou Xia, Zhepei Wei, Wei-Lin Chen, Danqi Chen, and Yu Meng. The surprising effectiveness of negative reinforcement in llm reasoning.arXiv preprint arXiv:2506.01347, 2025. 15 Appendix A Training and Evaluation Details In this section, we provide the comprehensive hyperparameter configurations used for training and evaluating. To ensure a st...

    work page arXiv 2025

  45. [45]

    If πθ < π lower, the gradient is nullified to prevent the probability from dropping too low (over-penalization)

    Lower Clip: πlower = (1 −ϵ low)πold. If πθ < π lower, the gradient is nullified to prevent the probability from dropping too low (over-penalization)

    work page

  46. [46]

    1 G GX i=1 ˆAi 1 sg(πθ(oi|q)) 1 |oi | ∇θ πθ(oi|q) 1 |oi | # =E q,{oi}

    Dual Clip: πupper = c·π old. If πθ > π upper, the gradient vanishes to prevent the policy from moving further away from the reference when the action is already deemed “wrong.” In the negative regime, the goal of optimization is todecreasethe probability of the token. Thus, the Capacity represents the maximum allowable decrease before the policy hits the ...

    work page