Pith. sign in

REVIEW 3 major objections 5 minor 27 references

When a trust region is measured by divergence, mask tokens by whether the next gradient step grows that divergence, not by the sampled importance ratio.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 08:48 UTC pith:D5HYECPF

load-bearing objection Clean closed-form fix to DPPO’s mismatched direction criterion; modest but consistent gains, with the first-order caveat already flagged by the authors. the 3 major comments →

arxiv 2607.10848 v1 pith:D5HYECPF submitted 2026-07-12 cs.LG

Predictive Divergence Masks for LLM RL

Xiangxin Zhou , Jiarui Yao , Penghui Qi , Bowen Ping , Jiaqi Tang , Haonan Wang , Tianyu Pang This is my paper
classification cs.LG
keywords LLM reinforcement learningtrust-region maskspredictive divergence maskdirectional derivativetop-K KLoff-policy policy gradientsoftmax policies
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

In reinforcement learning for large language models, updates are kept inside a trust region by masking tokens that would push the training policy too far from the behavior policy that produced the data. Prior work already replaced the usual proximity test (how far is the policy?) with a distributional divergence, but still decided the direction of the update from a single sampled token's importance ratio. This paper shows that that ratio can disagree in sign with the true change of the divergence, so the mask can keep updates that make the trust-region violation worse. It replaces the direction test with a closed-form first-order prediction of how the same divergence will move under the next policy-gradient step, and gives two cheap top-K estimators that work with the truncated probability lists that production rollouts actually return. Token-level checks and training runs then show that the new direction criterion tracks realized divergence change better than the ratio, and that the resulting mask improves stability and accuracy across model sizes and numerical precisions.

Core claim

For softmax policies, the first-order change of forward KL under a policy-gradient step has a closed form that splits into a local sampled-token gap (exactly the quantity the ratio-based direction test already uses) and a global softmax-coupling term that a single sample cannot see. Masking when that full coefficient has the same sign as the advantage, and the divergence already exceeds the threshold, yields a predictive divergence mask that is better aligned with the actual post-update change of the divergence than the inherited ratio-based direction criterion.

What carries the argument

The predictive divergence mask (Eq. 13): zero the token update when sign(Â · Ḋ) > 0 and D > δ, where Ḋ is the directional derivative of top-K KL along the unit policy-gradient direction, estimated with either an aggregated-tail or uniform-tail model of the unseen vocabulary mass.

Load-bearing premise

The sign of a local first-order derivative for one token is assumed to predict whether the real multi-token batch update will raise or lower that token's divergence.

What would settle it

On held-out rollouts, measure the fraction of tokens outside the trust region where the predictive mask and the ratio mask disagree; if the predictive mask does not show a lower rate of kept-but-diverged updates (ΔD > 0 after the actual step) and does not improve final task accuracy relative to the same proximity criterion with the ratio direction, the central claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper identifies an inconsistency in divergence-based trust-region masks for LLM RL (e.g., DPPO): the proximity criterion is distributional (top-K KL), but the direction criterion remains the PPO-style sampled-token ratio sign(Â(r−1)). It proposes the predictive divergence mask, which instead masks when sign(·Ḋ)>0 and D>δ, where Ḋ is the closed-form first-order directional derivative of D_KL(μ∥π) along the softmax policy-gradient step (Prop. 1 / Eq. 7). Two lightweight top-K estimators (aggregated-tail and uniform-tail) adapt Ḋ to truncated rollout statistics. Token-level analysis (61 seeds) reports better alignment of Ḋ with realized one-step ΔD than the ratio criterion on disagreement tokens, and training runs on Qwen3-4B/8B/30B (including FP8 regimes) show improved AIME avg@16 stability and accuracy over DPPO-TopK-KL and GRPO clip-higher.

Significance. If the result holds, the paper cleanly separates proximity and direction in trust-region masks and supplies a distributionally consistent direction criterion with a closed-form softmax derivation and practical top-K estimators. That is a genuine conceptual upgrade over inheriting PPO’s ratio direction after upgrading proximity to a divergence. Strengths include a clean derivation (Appendix B), the binary-KL reduction check (Appendix C), a TV extension (Appendix D), and multi-scale experiments plus a token-level mechanism study with many seeds. The contribution is incremental relative to DPPO but well-scoped and useful for production LLM RL, where top-K logprobs are the only available distributional signal.

major comments (3)
  1. [§4.2, Figs. 3–4] §4.2 and Figs. 3–4: the mechanism evidence is modest and concentrated on a small set. On average only ~82 of 1435 out-of-region tokens per seed disagree (~5.7%), and the unsafe-keep rate improves by 2.7 pp (36.9%→34.2%). The paper’s central hypothesis is that better directional alignment explains the training gains in Figs. 1–2. Please reconcile the small disagreement set and modest alignment gain with the larger accuracy/stability improvements, or provide additional evidence (e.g., cumulative effect over many steps, which tokens matter, or ablation that isolates only disagreement tokens).
  2. [Remarks 2–3, §3.2–3.4] Remarks 2–3 and §3.2–3.4: the mask uses the sign of a local first-order Ḋ under the assumptions that (i) the O(η²) remainder does not flip the sign at training step sizes and (ii) simultaneous multi-token batch updates do not reverse the local per-token ΔD. The §4.2 check is a one-step, same-batch comparison and does not fully stress multi-minibatch / multi-epoch dynamics. Either strengthen the validation (e.g., multi-step realized ΔD under the actual optimizer schedule) or state more clearly as a limitation that the predictive mask is a local heuristic whose sign can be reversed by cross-token interactions.
  3. [§4.1, Fig. 1] Fig. 1 and §4.1: main training curves appear to be single-run (or few-run) trajectories, while the token analysis uses 61 seeds with mean±SE. Without seed-level variance or statistical tests on AIME avg@16, it is hard to judge whether the predictive-mask gains over DPPO-TopK-KL are reliable, especially on the 30B FP8 settings where GRPO collapses. Please report multi-seed means/variance for the main comparison or justify why single-run curves suffice.
minor comments (5)
  1. [§3.2, Eq. (8)–(9)] Eq. (8) and the surrounding text correctly note that the ratio criterion is the local term; a short explicit statement that sign(π_k−μ_k)=sign(r−1) only when μ_k>0 (already implicit) would avoid edge-case confusion for near-zero behavior probabilities.
  2. [Fig. 2] Fig. 2 omits the uniform-tail curve present in Fig. 1; either include it for consistency or note why only aggregated-tail is shown at δ=0.05.
  3. [Appendix E, §4.1] Appendix E training-dynamics figures (5–8) are useful; a one-sentence pointer in §4.1 to lower clip fractions under predictive masks would help readers connect dynamics to the main claim.
  4. [Appendix A] Related work (Appendix A) is thorough on trust-region and DPPO lineage; a brief mention of how CISPO-style weight truncation differs from hard masking would further situate the contribution.
  5. [§3.1–3.2] Notation: Ḋ is introduced as the unit-direction derivative and then used with sign(·Ḋ); a single clarifying sentence that the mask never needs η or |r| (Remark 2) earlier in §3.1 would improve readability.

Circularity Check

1 steps flagged

No construction-level circularity: closed-form Ḋ is independent softmax calculus; DPPO self-citation supplies only the baseline proximity setup, not the new direction criterion or empirical claims.

specific steps
  1. self citation load bearing [§2 (DPPO baseline) and §3.4 (mask definition)]
    "DPPO (Qi et al., 2026) keeps this masked structure and its asymmetric direction criterion, but rebuilds the proximity criterion around the full divergence... Instantiating the mask of Eq. (4) with this top-K divergence... gives the DPPO-TopK-KL method that we take as the starting point and primary baseline"

    The proximity half of the trust-region mask and the experimental baseline are taken wholesale from prior work by overlapping authors. This is ordinary incremental research rather than a definitional loop, because the paper’s new contribution (the closed-form direction criterion and its empirical alignment) is derived and measured independently of that citation; the self-citation therefore contributes only a minor, non-forcing dependency.

full rationale

The load-bearing derivation (Prop. 1 / Eq. 7 and Appendix B) obtains the directional derivative of D_KL(μ∥π) along the unit policy-gradient direction v_i = 1[i=k]−π_i by ordinary differentiation of the softmax (Lemma 4) and the definition of forward KL; the local/global split then follows algebraically and is not defined in terms of the later mask or the empirical ΔD. The predictive mask (Eq. 13) simply substitutes sign(·Ḋ) for the inherited ratio sign; nothing in that substitution is fitted to the AIME or post-update data that later evaluate it. The two top-K tail estimators are likewise closed-form reductions of the same expression, not parameters tuned to the reported metrics. Heavy citation of DPPO (same author group) is used only to inherit the top-K KL proximity criterion and the experimental baseline; the paper’s novel claim—that the divergence-based direction better tracks realized ΔD and improves training—is tested against held-out accuracy and one-step ΔD measurements, not forced by the citation. Remarks 2–3 openly flag the first-order/local approximation as an assumption rather than a tautology. Consequently the derivation chain does not reduce to its inputs by construction; the single minor self-citation does not raise the score above 1.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 2 invented entities

The central claim rests on standard softmax RL math plus a few domain modeling choices (top-K truncation, first-order sign prediction, fixed δ/K). No new physical entities. Free parameters are ordinary RL hyperparameters; the load-bearing modeling choices are the first-order local approximation and the tail models for truncated vocabularies.

free parameters (4)
  • divergence threshold δ = 0.15 (main); 0.05 (sensitivity)
    Hand-chosen trust-region radius; main results use recommended δ=0.15, sensitivity uses δ=0.05. Mask decisions and reported gains depend on this threshold.
  • top-K support size K = 20
    Truncation width for both proximity D and Ḋ estimators; fixed at 20 for all divergence methods.
  • learning rate and batching = lr=1e-6; batch 64–128; group 8–16
    Shared training hyperparameters (lr 1e-6, batch/minibatch/group sizes per model) that determine the effective step size η underlying the first-order approximation.
  • GRPO clip-higher bounds = 0.2 / 0.28
    Baseline-only clip window (ε_low=0.2, ε_high=0.28); not used by the proposed mask but affects comparative claims.
axioms (5)
  • domain assumption Autoregressive LLM policies are discrete softmax distributions over a large vocabulary, so logit gradients take the form 1[i=k]−π_i.
    Used throughout §3.2 and Appendix B to obtain the closed-form Ḋ.
  • ad hoc to paper The sign of the first-order term in D(πη) ≈ D + η Ḋ dominates higher-order O(η²) remainder for the effective step sizes used in training.
    Remark 2; required for the predictive mask to track realized ΔD.
  • domain assumption Production rollouts expose only top-K log-probs plus tail mass, and top-K typically captures >99% of probability mass so D_TopK_KL is a tight lower bound.
    §2 and §3.3; inherited from DPPO and used to justify both proximity and the two Ḋ estimators.
  • standard math Forward KL controls total variation via Pinsker’s inequality, justifying D=D_KL as the trust-region proximity measure.
    Stated in §2 when choosing D_KL(μ∥π).
  • domain assumption Token-level advantages Â_t are given by the underlying critic-free estimator (GRPO-style) and are not redesigned.
    §2; isolates the contribution to the mask’s direction criterion.
invented entities (2)
  • predictive divergence mask M_pred independent evidence
    purpose: Replace sign(Â(r−1)) with sign(·Ḋ) while keeping the divergence proximity test D>δ.
    Core proposed object (Eq. 13); defined operationally from existing quantities, not a physical entity.
  • aggregated-tail and uniform-tail estimators of Ḋ independent evidence
    purpose: Estimate the directional derivative from truncated top-K rollout statistics without a full-vocabulary pass.
    Eqs. 10–11; two modeling choices for the unseen tail, shown to behave similarly when top-K mass is large.

pith-pipeline@v1.1.0-grok45 · 21228 in / 3511 out tokens · 51730 ms · 2026-07-14T08:48:12.498254+00:00 · methodology

0 comments
read the original abstract

Reinforcement learning for large language models (LLMs) typically relies on trust-region masks to stabilize off-policy updates. The dominant PPO-style approach uses the sampled-token importance ratio for two criteria: a proximity criterion, which asks whether the policy has moved too far from the behavior policy, and a direction criterion, which asks whether the update pushes it farther away. Recent work DPPO improves the proximity criterion by replacing PPO's ratio-based test with a probability divergence between the behavior and training policies. However, its direction criterion is still inherited from PPO. A token can be masked only when the sampled-token importance ratio moves away from one. We observe that this ratio-based direction criterion is a single-sample proxy that can disagree in sign with the change of the divergence that defines the proximity criterion. We therefore propose the predictive divergence mask, which asks whether the next policy-gradient step will increase or decrease the same divergence used by the trust region. For the discrete softmax policies used in LLM RL, we derive this prediction in closed form. Because production rollout engines expose only a truncated (top-K) view of the vocabulary, we develop two lightweight top-$K$ estimators for this prediction. Detailed analysis shows the divergence-based direction is better aligned with the realized change of the divergence than the sampled ratio, and the resulting masks improve RL training across model scales and precision settings.

Figures

Figures reproduced from arXiv: 2607.10848 by Bowen Ping, Haonan Wang, Jiaqi Tang, Jiarui Yao, Penghui Qi, Tianyu Pang, Xiangxin Zhou.

Figure 1
Figure 1. Figure 1: Evaluation accuracy (avg@16, averaged over AIME24 and AIME25) over training at [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Evaluation accuracy (avg@16, averaged over AIME24 and AIME25) across the four settings [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Unsafe-keep rate on disagreement tokens (δ = 0.15, 61 seeds). We consider tokens outside the trust region (D > δ) on which the ratio-based and divergence-based direction criteria make opposite keep/clip decisions. For each criterion, among the disagreement tokens it keeps, we report the fraction whose divergence increases after the update (∆D > 0). Lower is better. (a) Mean unsafe-keep rate across seeds. T… view at source ↗
Figure 4
Figure 4. Figure 4: Realized divergence change of kept disagreement tokens (δ = 0.15, 61 seeds). For each direction criterion, we collect the disagreement tokens it keeps and measure ∆D = Dpost − Dpre after one actual update. Tokens from all seeds are combined in this figure. (a) Empirical CDF of ∆D. The shaded region ∆D < 0 indicates kept updates that contracted the divergence. (b) Fraction of kept updates with ∆D < 0. The d… view at source ↗
Figure 5
Figure 5. Figure 5: Training dynamics for Qwen3-4B-Base at K = 20 and δ = 0.15. AIME24 and AIME25 are shown separately. The remaining panels report reward, response length, PPO-KL, and clip fraction. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Training dynamics for Qwen3-8B-Base at K = 20 and δ = 0.15, with the same metrics and methods as [PITH_FULL_IMAGE:figures/full_fig_p021_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Training dynamics for Qwen3-30B-A3B-Base under FP8 E2E training at [PITH_FULL_IMAGE:figures/full_fig_p021_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Training dynamics for Qwen3-30B-A3B-Base with FP8 Rollout at [PITH_FULL_IMAGE:figures/full_fig_p022_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

27 extracted references · 16 linked inside Pith

  1. [1]

    Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms.arXiv preprint arXiv:2402.14740,

    Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms.arXiv preprint arXiv:2402.14740,

    Pith/arXiv arXiv

  2. [2]

    Minimax-m1: Scaling test-time compute efficiently with lightning attention.arXiv preprint arXiv:2506.13585,

    Aili Chen, Aonian Li, Bangwei Gong, Binyang Jiang, Bo Fei, Bo Yang, Boji Shan, Changqing Yu, Chao Wang, Cheng Zhu, et al. Minimax-m1: Scaling test-time compute efficiently with lightning attention.arXiv preprint arXiv:2506.13585,

    Pith/arXiv arXiv

  3. [3]

    Deepseek-r1: Incentivizing reasoning capability in llms 11 via reinforcement learning.arXiv preprint arXiv:2501.12948,

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms 11 via reinforcement learning.arXiv preprint arXiv:2501.12948,

    Pith/arXiv arXiv

  4. [4]

    https://thinkingmachines.ai/blog/defeating-nondeterminism-in- llm-inference/

    doi: 10.64434/tml.20250910. https://thinkingmachines.ai/blog/defeating-nondeterminism-in- llm-inference/. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. InProceedings of the ACM SIGOPS 29th Sympo...

    doi:10.64434/tml.20250910

  5. [5]

    When speed kills stability: Demystifying rl collapse from the inference-training mismatch, 2025a

    Jiacai Liu, Yingru Li, Yuqian Fu, Jiawei Wang, Qian Liu, and Yu Shen. When speed kills stability: Demystifying rl collapse from the inference-training mismatch, 2025a. https://yingru.notion.site/When-Speed-Kills-Stability-Demystifying-RL-Collapse-from- the-Inference-Training-Mismatch-271211a558b7808d8b12d403fd15edda. Zichen Liu, Changyu Chen, Wenjun Li, P...

    Pith/arXiv arXiv

  6. [6]

    Beyond uniform token-level trust region in llm reinforcement learning.arXiv preprint arXiv:2606.10968,

    Renjie Mao, Xiangxin Zhou, Lvfang Tao, Yixin Ding, Yu Shi, Yongguang Lin, Yuheng Wu, Honglin Zhu, Qian Qiu, and Wenxi Zhu. Beyond uniform token-level trust region in llm reinforcement learning.arXiv preprint arXiv:2606.10968,

    Pith/arXiv arXiv

  7. [7]

    Defeating the training-inference mismatch via fp16.arXiv preprint arXiv:2510.26788,

    Penghui Qi, Zichen Liu, Xiangxin Zhou, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Defeating the training-inference mismatch via fp16.arXiv preprint arXiv:2510.26788,

    arXiv

  8. [8]

    Rethinking the trust region in llm reinforcement learning.arXiv preprint arXiv:2602.04879,

    Penghui Qi, Xiangxin Zhou, Zichen Liu, Tianyu Pang, Chao Du, Min Lin, and Wee Sun Lee. Rethinking the trust region in llm reinforcement learning.arXiv preprint arXiv:2602.04879,

    Pith/arXiv arXiv

  9. [9]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

    Pith/arXiv arXiv

  10. [10]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    Pith/arXiv arXiv

  11. [11]

    Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv: 2409.19256,

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv: 2409.19256,

    Pith/arXiv arXiv

  12. [12]

    Megatron-lm: Training multi-billion parameter language models using model parallelism

    12 Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053,

    Pith/arXiv arXiv 1909

  13. [13]

    Kimi k1.5: Scaling reinforcement learning with llms

    Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1.5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599, 2025a. Ling Team, Bin Han, Caizhi Tang, Chen Liang, Donghao Zhang, Fan Yuan, Feng Zhu, Jie Gao, Jingyu Hu, Longfei Li, Meng Li, Mingyang ...

    Pith/arXiv arXiv

  14. [14]

    Simple policy optimization

    Zhengpeng Xie, Qiang Zhang, Fan Yang, Marco Hutter, and Renjing Xu. Simple policy optimization. arXiv preprint arXiv:2401.16025,

    Pith/arXiv arXiv

  15. [15]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    Pith/arXiv arXiv

  16. [16]

    Jiarui Yao, Xiangxin Zhou, Penghui Qi, Wee Sun Lee, Liefeng Bo, and Tianyu Pang

    https://fengyao.notion.site/off- policy-rl. Jiarui Yao, Xiangxin Zhou, Penghui Qi, Wee Sun Lee, Liefeng Bo, and Tianyu Pang. Rethinking the divergence regularization in llm rl.arXiv preprint arXiv:2606.09821,

    Pith/arXiv arXiv

  17. [17]

    Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476,

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476,

    Pith/arXiv arXiv

  18. [18]

    Stabilizing reinforcement learning with llms: Formulation and practices.arXiv preprint arXiv:2512.01374,

    Chujie Zheng, Kai Dang, Bowen Yu, Mingze Li, Huiqiang Jiang, Junrong Lin, Yuqiong Liu, Hao Lin, Chencan Wu, Feng Hu, et al. Stabilizing reinforcement learning with llms: Formulation and practices.arXiv preprint arXiv:2512.01374,

    arXiv

  19. [19]

    Reinforcing general reasoning without verifiers.arXiv preprint arXiv:2505.21493, 2025a

    Xiangxin Zhou, Zichen Liu, Anya Sims, Haonan Wang, Tianyu Pang, Chongxuan Li, Liang Wang, Min Lin, and Chao Du. Reinforcing general reasoning without verifiers.arXiv preprint arXiv:2505.21493, 2025a. 13 Xiangxin Zhou, Zichen Liu, Haonan Wang, Chao Du, Min Lin, Chongxuan Li, Liang Wang, and Tianyu Pang. Variational reasoning for language models.arXiv prepr...

    Pith/arXiv arXiv

  20. [20]

    and CPO (Achiam et al., 2017), which enforce an explicit KL constraint, and of PPO (Schulman et al., 2017), which approximates it with the cheap clipped surrogate of Eq. (2). Mirror-descent and proximal formulations (Tomar et al., 2022; Wang et al., 2020,

    2017

  21. [21]

    GRPO (Shao et al.,

    and modern reasoning training (Guo et al., 2025; Team et al., 2025a; Zhou et al., 2025a,b). GRPO (Shao et al.,

    2025

  22. [22]

    removes the critic with group-relative advantages, and recent follow-ups (Ahmadian et al., 2024; Liu et al., 2025b; Yu et al., 2025; Chen et al.,

    2024

  23. [23]

    These methods modify the objective or the clipping rule, but they still decide the update direction from the sampled importance ratio

    truncates importance weights instead of masking. These methods modify the objective or the clipping rule, but they still decide the update direction from the sampled importance ratio. In deployed LLM RL systems, an additional source of instability is the training-inference mismatch. The rollout policy served by an optimized inference engine can differ fro...

    2025

  24. [24]

    (5) that we build on

    reinterprets PPO’s clip as a one-sample estimate of total variation and replaces it with an explicit divergence mask, including the top-K KL realization of Eq. (5) that we build on. Complementary work studies non-uniform token-level trust regions that vary the trust-region constraint across tokens (Mao et al., 2026). Other divergence-based methods (Luo et...

    2026

  25. [25]

    Aggregated-tail estimator.Under the top-K aggregated-tail construction, the support contains the retained tokensK and one tail bucket

    On any fixed support, Lemma 4 gives the directional derivative of TV along the unit policy-gradient direction: ˙DTV =− 1 2 X i si πi vi − X j πjvj ,wheres i = sign(µi −π i).(34) The only difference from the KL case is the factorsi = sign(µi −π i), which comes from differentiating the absolute value in TV. Aggregated-tail estimator.Under the top-K aggregat...

    2024

  26. [26]

    By default, both training and rollout use BF16

    for rollout generation. By default, both training and rollout use BF16. For Qwen3-30B-A3B-Base, we evaluate two low-precision settings: FP8 only in vLLM (FP8 Rollout) and FP8 in both Megatron-LM and vLLM (FP8 E2E). Hyperparameters.All methods use the same learning rate, batching, rollout temperature, response budget, and group size. Each rollout batch is ...

    2024

  27. [27]

    0 100 200 300 Training Step 0.2 0.4 0.6Reward Training Reward 0 100 200 300 Training Step 0.0 0.1 0.2 0.3 0.4Accuracy AIME 2024 0 100 200 300 Training Step 0.0 0.1 0.2 0.3Accuracy AIME 2025 0 100 200 300 Training Step 0 2000 4000 6000T okens Response Length 0 100 200 300 Training Step 0.00 0.01 0.02 0.03 0.04KL PPO-KL 0 100 200 300 Training Step 10−3 10−2...

    2024