Pith. sign in

REVIEW 4 major objections 7 minor 63 references

ARMOR claims that injecting correct reference-policy samples and optimizing a mixed policy prevents the validation collapse that standard KL regularization cannot stop in long-horizon LLM reasoning RL.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 07:13 UTC pith:XE2G3UOS

load-bearing objection ARMOR is a plausible fix for late-training validation collapse, with a cute trust-region derivation, but the main empirical comparison is confounded by a silent query filter in the rollout algorithm and needs better controls before the headline numbers are credible. the 4 major comments →

arxiv 2607.10481 v2 pith:XE2G3UOS submitted 2026-07-11 cs.LG cs.AIcs.CL

ARMOR: Stabilizing On-Policy LLM RL with Off-Policy Anchor Samples

Kexin Huang , Junkang Wu , Jinda Lu , Shuo Yang , Chiyu Ma , Jiancan Wu , Xiang Wang , Xiangnan He
show 2 more authors
Guoyin Wang Jingren Zhou
This is my paper
classification cs.LG cs.AIcs.CL
keywords reinforcement learningLLM reasoningover-optimizationreverse KL regularizationvalidation collapseanchor rolloutmixed policy optimizationadaptive trust region
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Long-horizon reinforcement learning for LLM reasoning can show steadily rising training reward while validation accuracy falls—a failure the paper calls over-optimization. The paper argues the standard defense, reverse KL regularization, is structurally insufficient: it is mode-seeking, so the policy can collapse onto a narrow set of high-reward patterns without penalty, and it applies a uniform penalty that, if strengthened, stalls exploration. ARMOR replaces the passive penalty with active stabilization: each training batch includes a correct sample generated by the reference policy (Anchor Rollout), and the policy objective is rewritten to optimize a mixture alpha*pi_theta + (1-alpha)*pi_ref (Mixed Optimization), which yields an adaptive trust region. The paper reports that this combination prevents validation collapse and sustains or improves reasoning scores on math benchmarks beyond the baseline's collapse point. A sympathetic reader would take the central claim to be that preserving known-good solution patterns through sampling, rather than penalizing drift, is the missing ingredient for stable RL scaling.

Core claim

The central claim is that validation collapse in reinforcement learning with verifiable rewards is not primarily a reward-design problem but an algorithmic distributional one: the policy drifts off the reference distribution's generalizable modes even when rewards are verifiable. ARMOR counters this by injecting, for every query, one correct off-policy sample drawn from the reference policy via rejection sampling, ensuring the model keeps seeing established correct solutions; and by replacing the standard importance-sampling ratio with r_mix = (alpha*pi_theta + (1-alpha)*pi_ref) / (alpha*pi_theta_old + (1-alpha)*pi_ref). The paper proves that gains on the mixture target strictly transfer to

What carries the argument

The load-bearing object is the mixed importance-sampling ratio r_mix = (alpha*pi_theta + (1-alpha)*pi_ref) / (alpha*pi_theta_old + (1-alpha)*pi_ref), which makes the data distribution generated by Anchor Rollout—a mixture of on-policy samples and one correct anchor sample per query—consistent with the optimization target. Solving r_mix = 1 +/- epsilon gives the effective trust-region boundary pi_theta = (1 +/- epsilon)*pi_theta_old +/- epsilon*(1-alpha)/alpha*pi_ref; the second term is the adaptive expansion that lets updates be larger where the reference supports the action. The anchor sample itself is produced by rejection sampling from pi_ref with the requirement R(x, y_anc) = 1, so the r

Load-bearing premise

The method assumes the correct samples drawn from the reference checkpoint are genuinely generalizable solution patterns; if the reference has already over-optimized onto shortcut modes, the anchor will preserve exactly those modes, and the stabilization mechanism would hold the model at the wrong answer.

What would settle it

Use ARMOR from a checkpoint whose correct-answer samples are known to be over-optimized shortcuts (a model trained past the validation peak), and compare validation trajectories against ARMOR from a clean peak checkpoint. If the anchor-sourced correct samples do not restore generalization, then keeping the reference's correct modes is not the operative mechanism; alternatively, run ARMOR with anchors sampled without the correctness filter and see whether validation collapse returns.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Standard reverse KL regularization cannot bridge the validation gap: with small coefficients it only delays collapse, and with a large coefficient it freezes the policy; this holds for both k2 and k3 KL estimators.
  • Because gains on the mixture objective strictly imply gains on pi_theta, optimizing the mixed policy is a valid way to improve the target policy rather than a proxy that can diverge from it.
  • The clipping boundary expands where pi_ref assigns probability, so correct reference-supported tokens can receive larger positive updates and reference-favored wrong tokens stronger negative updates, without a uniform exploration penalty.
  • Periodic reset of the reference policy to the current policy prevents the anchor from becoming a fixed bottleneck; a static anchor stabilizes but caps the ceiling.
  • ARMOR can rescue a model already in a collapsed state, but starting from the best pre-collapse checkpoint yields a higher final ceiling.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same mechanism should transfer to coding or agentic tasks with verifiable rewards: if correct reference traces are injected and the objective is mixed, validation collapse from shortcut exploitation should be delayed there as well—a direct test of the paper's generality.
  • The rejection-sampling filter (only correct anchors) is probably what makes Anchor Rollout work; a natural control experiment would swap in unlabeled reference samples. If unlabeled anchors work equally well, the paper's explanation in terms of preserving correct modes would need revision.
  • The adaptive trust-region view suggests a parameter-free alternative: one could set alpha from the estimated overlap between pi_theta and pi_ref, so the expansion term grows exactly when the policy has drifted, instead of tuning alpha per model.
  • The paper's own limitation about computational overhead points to an offline replay buffer of high-quality anchors; if such a buffer preserves the benefit, the method becomes substantially cheaper and more practical.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper studies the over-optimization failure mode in reinforcement learning with verifiable rewards (RLVR) for LLM reasoning and proposes ARMOR. Section 3 presents an empirical study in a continual-training setup (DAPO on Qwen2.5-Math-7B, resumed from the best AIME24 checkpoint): reverse-KL penalties at β ≤ 0.1 delay but do not prevent validation collapse, while β = 1.0 stabilizes only by inducing stagnation, and a forward/reverse KL asymmetry is used to explain this. The authors conclude that passive KL regularization is structurally insufficient. ARMOR replaces the KL penalty with two components: (1) Anchor Rollout, which injects one reward-correct sample from a reference policy πref (initialized as the run's best checkpoint) into each response group via rejection sampling over four rollouts (Algorithm 1); and (2) Mixed Optimization, which uses the mixture IS ratio r_mix = (απθ + (1−α)πref)/(απθ_old + (1−α)πref), interpreted as an adaptive trust region (Eq. 6) whose gains on the mixture target are claimed to transfer to πθ (Sec. 4.2). Experiments on Qwen2.5-Math-7B and Qwen3-8B-Base with DAPO and QAE report gains on AIME24/25 and AMC while maintaining general-task performance (Table 1), supported by ablations of the two components, reference-reset frequency, and initialization quality. The central claim is that ARMOR prevents validation collapse and lifts the performance ceiling over extended training horizons.

Significance. If the empirical identification were clean, this would be a solid and practical contribution. The reverse-KL insufficiency analysis (Sec. 3) is informative, including the k2/k3 estimator checks and the forward-vs-reverse KL asymmetry; the algebra in Eq. (6) is correct; and the 'strictly translates' claim in Sec. 4.2 is actually the exact linear relation Δmix = αΔθ, which strengthens the argument. The paper ships its code, evaluates on external benchmarks, and its ablations (Figs. 3-4) directly probe the two proposed components; the Limitations section honestly acknowledges the added hyperparameters and the lack of convergence guarantees. However, the empirical core is currently confounded: Algorithm 1 introduces a query-level selection gate that no ablation isolates, and the headline results are single runs with checkpoints selected on the reported AIME24 metric. The contribution is therefore conditional on closing this identification gap.

major comments (4)
  1. [Algo. 1, Line 13 (Sec. 4.1, 5.2)] The batch condition adds a query-level gate beyond DAPO's dynamic-sampling rule: a query is kept only if max(R_off|x)=1, i.e., the reference policy obtains at least one correct response within the 4-sample budget. Because πref is the run's own best checkpoint, this silently removes questions the current best model cannot solve, biasing the training distribution. The manuscript gives no statistics on the dropped-query fraction or its correlation with difficulty. Figs. 3a/3b compound this: every variant that includes anchors also includes the gate, and every variant without anchors removes both. Hence the reported gains over DAPO cannot be causally attributed to anchor injection or the mixed ratio until a control (e.g., DAPO with the identical gate and standard IS ratio but no anchor) and the gate statistics are provided.
  2. [Table 1, Sec. 5.1, Fig. 6] All reported checkpoints are selected on the AIME24 validation metric, so the headline numbers are post-hoc maxima over the training trajectory, and each is a single run without error bars. The QAE re-run in App. C.3 (Fig. 6) shows run-to-run spread that cannot be assessed for the primary DAPO comparisons; the +11.15 gain on Qwen3 is one trajectory. The phrase 'significantly higher performance ceiling' (Sec. 4.2) and the abstract's claims require at least 2-3 seeds with mean±std or per-run curves for the headline configurations, and ideally metrics at matched step counts in addition to metric-selected checkpoints.
  3. [Sec. 4.2, Eq. (5) vs. Eq. (3), App. B] The mixture model treats the anchor as drawn from πref, but Eq. (3) specifies y_anc ~ πref(·|x) conditioned on R=1, with rejection sampling and argmax over up to 4 attempts; πmix in Eq. (4) ignores this conditioning, so r_mix in Eq. (5) is not the exact likelihood ratio for anchor tokens. Also, the group is |G|−1 on-policy responses plus one anchor (App. B keeps |G|=16), so the natural α for Qwen2.5 is 15/16, which is used, but Qwen3 uses α=0.875, inconsistent with that composition. Please correct the ratio for the conditional anchor distribution or state explicitly that Eq. (5) is a heuristic approximation, and quantify the difference relative to the standard off-policy IS variant in Fig. 3b.
  4. [Sec. 4.1, Eq. (3), Fig. 3c] The mechanism's premise is that injected samples are 'established generalizable solution patterns.' Since πref is initialized from the best checkpoint of the same DAPO run (Sec. 3 footnote; Sec. 4.1), it may contain exactly the over-optimized modes that cause validation collapse, and rejection sampling on the training reward cannot separate generalizable from over-optimized correct solutions. Fig. 3c varies only the checkpoint used for initialization; it does not test anchor content. A diagnostic (e.g., validation accuracy of the anchor solutions, or comparison of anchors from the collapsed vs. stable checkpoint) is needed, because this assumption is load-bearing for the 'active mode retention' story.
minor comments (7)
  1. [Footnote 3] The citation 'Liu et al., 2025a' is ambiguous: the bibliography contains two distinct Liu et al. 2025a entries (Aixin Liu et al., DeepSeek-V3.2, and Jiacai Liu et al.). Please disambiguate.
  2. [Sec. 4.2] The 'strictly translates' implication is in fact an equivalence: with πref fixed within a step, Δmix = αΔθ exactly. State this directly; it strengthens rather than weakens the argument.
  3. [App. B] The group-size arithmetic should be clarified in one place: Eq. (3) implies 15 on-policy + 1 anchor = 16 responses, which matches α=0.9375 for Qwen2.5 but not α=0.875 for Qwen3. Specify whether the anchor replaces or appends to the on-policy set, and how α is chosen for each model.
  4. [Table 1] The 'Step' column semantics are unclear: for instance '+200' presumably means 200 steps after the best checkpoint (i.e., step 340 overall). Please state this explicitly so training budgets are comparable.
  5. [App. A] The differentiation from the concurrent experience-guided preprints (EAPO, OWPO, FIPO) is purely conceptual. Given the shared authorship cluster, add at least a brief algorithmic comparison or a quantitative side-by-side result to substantiate the claimed difference from 'bootstrap a new model.'
  6. [Limitations] The query-selection gate in Algo. 1, Line 13 is not discussed in the Limitations section. Given its confounding potential, it should be explicitly acknowledged and analyzed there.
  7. [References] Many reference entries truncate author lists to 'and 1 others'; please use the journal's full-author formatting.

Circularity Check

0 steps flagged

No significant circularity: ARMOR's core derivations are algebraic/linearity identities and its headline results are measured against external benchmarks; the many overlapping-author citations are related-work context, not load-bearing inputs.

full rationale

The paper's derivation chain does not reduce to its own inputs. Anchor Rollout and Mixed Optimization are defined operationally (Algo. 1, Eq. 3-5); the mixed IS ratio rmix is a ratio of mixture probabilities, and the adaptive trust-region boundary (Eq. 6) follows by direct algebra from setting rmix=1±ε. The 'strictly translates' claim in Sec. 4.2 is true by linearity of expectation (since πref is fixed, the mixture improvement equals α times the πθ improvement), so it is a trivial identity rather than a fitted or definitionally circular prediction. Section 3's KL-insufficiency claim is an empirical result under a controlled continual-training protocol, not an assumption built into ARMOR. The main tables compare against DAPO and QAE on external benchmarks (AIME/AMC/GPQA/MMLU-Pro), so the central empirical claim is self-contained against external evidence. The many same-group preprints (EAPO, FIPO, OWPO, QAE, etc.) are cited in related-work discussions and as a baseline, not as the justification for ARMOR's mechanism; no 'uniqueness' or load-bearing self-citation is invoked. The Limitations section explicitly leaves a theoretical convergence guarantee open, which further shows the trust-region analysis is not being presented as a closed derivation. The skeptic's query-filter confound (Algo. 1, Line 13 dropping queries with max(R_off|x)=0) is a potential selection/identification issue that is not reported, but it is not circularity: the filter changes the training distribution rather than encoding the predicted outcome. Hence no circular step is identifiable by the paper's own equations or self-citation chain.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

The central empirical claim rests on a few manually tuned hyperparameters and domain assumptions about what the reference policy represents, rather than on a closed-form derivation. The trust-region analysis is explanatory, not a proof. No new physical or model-level entities are postulated.

free parameters (3)
  • mixing coefficient alpha = 0.9375 (Qwen2.5-Math-7B), 0.875 (Qwen3-8B-Base)
    Controls the mixture pi_mix = alpha*pi_theta + (1-alpha)*pi_ref in Eq. 4 and the mixed IS ratio in Eq. 5; values are chosen per model, not derived, and directly determine the trust-region boundary in Eq. 6.
  • reference reset schedule tau = tau=160 for Qwen2.5-Math-7B; tau=1 for first 110/180 steps then fixed for Qwen3-8B-Base
    Manually determined per model and algorithm; the Limitations section states the reference reset strategy is determined manually. Reset frequency changes the anchor evolution and final performance (Fig. 4, Fig. 7).
  • rejection-sampling budget per query = 4 reference samples per query
    Used to obtain a correct anchor y_anc with R=1; if none of four reference samples is correct, no anchor is added. The effect of this budget on coverage is not swept.
axioms (5)
  • domain assumption Validation collapse in DAPO/QAE is caused by algorithmic over-optimization with loss of reference coverage, not by proxy-reward mismatch.
    Used to motivate the whole intervention in Sec. 1 and Sec. 3. If collapse were instead an evaluation artifact or reward issue, anchor samples would not target the cause.
  • domain assumption A correct sample from pi_ref is an established generalizable solution pattern whose injection preserves generalizable modes.
    Invoked in Sec. 4.1 Eq. 3. Correctness under the training reward (R=1) is assumed to imply generalizability; the reference itself may have already over-optimized.
  • standard math Reverse KL is mode-seeking and forward KL measures dropped modes; hence high forward KL indicates mode collapse.
    Standard properties of KL divergence; used in Sec. 3 analysis and Fig. 2b/2c.
  • standard math Optimizing against the mixture policy yields gains for pi_theta because the mixture expectation difference equals alpha times the pi_theta difference.
    Sec. 4.2: E_mix[R] - E_mix_old[R] = alpha*(E_pi_theta[R] - E_pi_theta_old[R]) by linearity; the claimed benefit to pi_theta, however, assumes the practical clipped objective approximates this expectation.
  • domain assumption Periodically resetting the reference keeps anchors from becoming stale; a manually chosen reset frequency is a valid design for the target settings.
    Sec. 5.2.3 and Appendix C.3 demonstrate empirical sensitivity to reset frequency; no principled criterion for reset timing is provided.

pith-pipeline@v1.3.0-alltime-deepseek · 3594 in / 4187 out tokens · 179919 ms · 2026-08-02T07:13:29.236714+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of ARMOR: Stabilizing On-Policy LLM RL with Off-Policy Anchor Samples." pith.science (2026). https://pith.science/paper/XE2G3UOS

@misc{pith2026260710481,
  author       = {Pith},
  title        = {Pith review of: ARMOR: Stabilizing On-Policy LLM RL with Off-Policy Anchor Samples},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XE2G3UOS}},
  note         = {Machine review of arXiv:2607.10481}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Reinforcement learning (RL) has significantly enhanced the reasoning capabilities of large language models (LLMs), yet the training process remains notoriously fragile. In this work, we investigate a critical source of this instability: over-optimization, where models exploit training heuristics at the expense of generalizable reasoning. While reverse KL regularization is the standard defense against such degradation, our analysis reveals that it is often insufficient in this regime, as it fails to ensure comprehensive coverage of the reference distribution. To address this, we propose ARMOR (Anchor Rollout and Mixed Optimization for RL), a framework that shifts the paradigm from passive penalty to active sample stabilization. ARMOR comprises two key components: (1) Anchor Rollout, which leverages off-policy data from the reference policy to preserve established solution patterns; and (2) Mixed Optimization, which reformulates the policy objective to enable controlled exploration without relying on auxiliary losses. Extensive experiments on reasoning benchmarks validate that ARMOR effectively mitigates validation collapse, enabling sustained performance improvements over extended training horizons.

Figures

Figures reproduced from arXiv: 2607.10481 by Chiyu Ma, Guoyin Wang, Jiancan Wu, Jinda Lu, Jingren Zhou, Junkang Wu, Kexin Huang, Shuo Yang, Xiangnan He, Xiang Wang.

Figure 1
Figure 1. Figure 1: Illustration of the over-optimization issue and [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: (a) Performance trends under KL regularization. Standard penalties (β ≤ 0.1: 1e-3, 1e-2, and 1e-1) merely delay validation collapse of the baseline, while the extreme setting (β = 1.0) ensures stability only by inducing stagnation, highlighting the stability-plasticity dilemma. (b) Asymmetry in KL divergence (measured after 100 steps of continual training). While increasing β drastically reduces the optimi… view at source ↗
Figure 3
Figure 3. Figure 3: (a) Necessity of Anchor Rollout. We compare DAPO against variants using only Mixed Optimization (α = 0.9, 0.8, and 0.7). Without the stability foundation provided by Anchor Rollout, these models suffer from eventual performance collapse. (b) Necessity of Mixed Optimization. We replace mixed optimization with alternative objectives on top of Anchor Rollout (AR). While AR alone effectively stabilizes trainin… view at source ↗
Figure 4
Figure 4. Figure 4: Impact of Reference Reset Frequency. We compare ARMOR training with a static anchor (no reset) versus resetting the reference policy every 80, 120, or 160 steps (noted by “Anchor Set” in the figure). Periodic resets allows the anchor to evolve with the policy, preventing stagnation and fostering higher reasoning accuracy. optimization objectives: (a) Standard DAPO: de￾fault objective with no modification t… view at source ↗
Figure 5
Figure 5. Figure 5: KL-regularized training performance (us￾ing k2 estimation). Consistent with k3 results (Fig. 2a), standard penalties (β ≤ 0.1: 1e-3, 1e-2, 1e-1) merely delay validation collapse. The extreme setting (β = 1.0) enforces stability but induces stagnation, validating the stability-plasticity dilemma in KL-regularization. • Qwen3-8B-Base: A two-stage strategy, start￾ing with τ = 1 for the first 180 steps, follow… view at source ↗
Figure 6
Figure 6. Figure 6: Hyperparameter robustness of ARMOR on QAE. We compare ARMOR variants against the [PITH_FULL_IMAGE:figures/full_fig_p014_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Ablation of Reset Strategy on Qwen3-8B-Base. Due to Qwen3’s strong capability, continuous reference [PITH_FULL_IMAGE:figures/full_fig_p015_7.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

63 extracted references · 27 linked inside Pith

  1. [1]

    arXiv preprint arXiv:2412.16720 , year=

    Openai o1 system card , author=. arXiv preprint arXiv:2412.16720 , year=

    Pith/arXiv arXiv

  2. [2]

    Nature , volume =

    DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning , author =. Nature , volume =

  3. [3]

    2: Pushing the frontier of open large language models , author=

    Deepseek-v3. 2: Pushing the frontier of open large language models , author=. arXiv preprint arXiv:2512.02556 , year=

    Pith/arXiv arXiv

  4. [4]

    arXiv preprint arXiv:2507.06261 , year=

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities , author=. arXiv preprint arXiv:2507.06261 , year=

    Pith/arXiv arXiv

  5. [5]

    arXiv preprint arXiv:2505.09388 , year=

    Qwen3 technical report , author=. arXiv preprint arXiv:2505.09388 , year=

    Pith/arXiv arXiv

  6. [6]

    5: Scaling reinforcement learning with llms , author=

    Kimi k1. 5: Scaling reinforcement learning with llms , author=. arXiv preprint arXiv:2501.12599 , year=

    Pith/arXiv arXiv

  7. [7]

    2024 , eprint=

    Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement , author=. 2024 , eprint=

    2024

  8. [8]

    arXiv preprint arXiv:2411.15124 , year=

    Tulu 3: Pushing frontiers in open language model post-training , author=. arXiv preprint arXiv:2411.15124 , year=

    Pith/arXiv arXiv

  9. [9]

    arXiv preprint arXiv:2503.24290 , year=

    Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model , author=. arXiv preprint arXiv:2503.24290 , year=

    Pith/arXiv arXiv

  10. [10]

    arXiv preprint arXiv:1707.06347 , year=

    Proximal policy optimization algorithms , author=. arXiv preprint arXiv:1707.06347 , year=

    Pith/arXiv arXiv

  11. [11]

    International conference on machine learning , pages=

    Trust region policy optimization , author=. International conference on machine learning , pages=. 2015 , organization=

    2015

  12. [12]

    arXiv preprint arXiv:1506.02438 , year=

    High-dimensional continuous control using generalized advantage estimation , author=. arXiv preprint arXiv:1506.02438 , year=

    Pith/arXiv arXiv

  13. [13]

    arXiv preprint arXiv:2402.03300 , year=

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models , author=. arXiv preprint arXiv:2402.03300 , year=

    Pith/arXiv arXiv

  14. [14]

    arXiv preprint arXiv:2503.14476 , year=

    Dapo: An open-source llm reinforcement learning system at scale , author=. arXiv preprint arXiv:2503.14476 , year=

    Pith/arXiv arXiv

  15. [15]

    arXiv preprint arXiv:2503.20783 , year=

    Understanding r1-zero-like training: A critical perspective , author=. arXiv preprint arXiv:2503.20783 , year=

    Pith/arXiv arXiv

  16. [16]

    arXiv preprint arXiv:2507.20673 , year=

    Geometric-Mean Policy Optimization , author=. arXiv preprint arXiv:2507.20673 , year=

    arXiv

  17. [17]

    2025 , eprint=

    Quantile Advantage Estimation for Entropy-Safe Reasoning , author=. 2025 , eprint=

    2025

  18. [18]

    arXiv preprint arXiv:2507.18071 , year=

    Group Sequence Policy Optimization , author=. arXiv preprint arXiv:2507.18071 , year=

    Pith/arXiv arXiv

  19. [19]

    Your Efficient RL Framework Secretly Brings You Off-Policy RL Training , url =

    Yao, Feng and Liu, Liyuan and Zhang, Dinghuai and Dong, Chengyu and Shang, Jingbo and Gao, Jianfeng , year =. Your Efficient RL Framework Secretly Brings You Off-Policy RL Training , url =

  20. [20]

    When Speed Kills Stability: Demystifying

    Liu, Jiacai and Li, Yingru and Fu, Yuqian and Wang, Jiawei and Liu, Qian and Jiang, Zhuo , year =. When Speed Kills Stability: Demystifying

  21. [21]

    2025 , eprint=

    Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routers , author=. 2025 , eprint=

    2025

  22. [22]

    2025 , eprint=

    Stabilizing Reinforcement Learning with LLMs: Formulation and Practices , author=. 2025 , eprint=

    2025

  23. [23]

    Small Leak Can Sink a Great Ship--Boost RL Training on MoE with IcePop! , url =

    Xin Zhao and Yongkang Liu and Kuan Xu and Jia Guo and Zihao Wang and Yan Sun and Xinyu Kong and Qianggang Cao and Liang Jiang and Zujie Wen and Zhiqiang Zhang and Jun Zhou , year =. Small Leak Can Sink a Great Ship--Boost RL Training on MoE with IcePop! , url =

  24. [24]

    Reward Hacking in Reinforcement Learning

    Weng, Lilian. Reward Hacking in Reinforcement Learning. lilianweng.github.io. 2024

    2024

  25. [25]

    International Conference on Machine Learning , pages=

    Scaling laws for reward model overoptimization , author=. International Conference on Machine Learning , pages=. 2023 , organization=

    2023

  26. [26]

    Junkang Wu and Kexin Huang and Xue Wang and Jinyang Gao and Bolin Ding and Jiancan Wu and Xiangnan He and Xiang Wang , booktitle=. Re. 2025 , url=

    2025

  27. [27]

    arXiv preprint arXiv:2510.22543 , year=

    FAPO: Flawed-Aware Policy Optimization for Efficient and Reliable Reasoning , author=. arXiv preprint arXiv:2510.22543 , year=

    arXiv

  28. [28]

    arXiv preprint arXiv:2509.04664 , year=

    Why language models hallucinate , author=. arXiv preprint arXiv:2509.04664 , year=

    Pith/arXiv arXiv

  29. [29]

    arXiv preprint arXiv:2509.09177 , year=

    Clip your sequences fairly: Enforcing length fairness for sequence-level rl , author=. arXiv preprint arXiv:2509.09177 , year=

    arXiv

  30. [30]

    arXiv preprint arXiv:2508.17850 , year=

    GEPO: Group Expectation Policy Optimization for Stable Heterogeneous Reinforcement Learning , author=. arXiv preprint arXiv:2508.17850 , year=

    arXiv

  31. [31]

    Advances in Neural Information Processing Systems (NeurIPS) , volume=

    Learning to summarize with human feedback , author=. Advances in Neural Information Processing Systems (NeurIPS) , volume=

  32. [32]

    Advances in neural information processing systems , volume=

    Training language models to follow instructions with human feedback , author=. Advances in neural information processing systems , volume=

  33. [33]

    arXiv preprint arXiv:2506.09477 , year=

    On a few pitfalls in KL divergence gradient estimation for RL , author=. arXiv preprint arXiv:2506.09477 , year=

    Pith/arXiv arXiv

  34. [34]

    arXiv preprint arXiv:2512.21852 , year=

    A Comedy of Estimators: On KL Regularization in RL Training of LLMs , author=. arXiv preprint arXiv:2512.21852 , year=

    arXiv

  35. [35]

    arXiv preprint arXiv:2510.20817 , year=

    KL-Regularized Reinforcement Learning is Designed to Mode Collapse , author=. arXiv preprint arXiv:2510.20817 , year=

    arXiv

  36. [36]

    Beyond Reverse

    Chaoqi Wang and Yibo Jiang and Chenghao Yang and Han Liu and Yuxin Chen , booktitle=. Beyond Reverse. 2024 , url=

    2024

  37. [37]

    First Conference on Language Modeling , year=

    Gpqa: A graduate-level google-proof q&a benchmark , author=. First Conference on Language Modeling , year=

  38. [38]

    Advances in Neural Information Processing Systems , volume=

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark , author=. Advances in Neural Information Processing Systems , volume=

  39. [39]

    2021 , eprint=

    Evaluating Large Language Models Trained on Code , author=. 2021 , eprint=

    2021

  40. [40]

    Does Reinforcement Learning Really Incentivize Reasoning Capacity in

    Yang Yue and Zhiqi Chen and Rui Lu and Andrew Zhao and Zhaokai Wang and Yang Yue and Shiji Song and Gao Huang , booktitle=. Does Reinforcement Learning Really Incentivize Reasoning Capacity in. 2025 , url=

    2025

  41. [41]

    arXiv preprint arXiv:2505.22617 , year=

    The entropy mechanism of reinforcement learning for reasoning language models , author=. arXiv preprint arXiv:2505.22617 , year=

    Pith/arXiv arXiv

  42. [42]

    Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for

    Shenzhi Wang and Le Yu and Chang Gao and Chujie Zheng and Shixuan Liu and Rui Lu and Kai Dang and Xiong-Hui Chen and Jianxin Yang and Zhenru Zhang and Yuqiong Liu and An Yang and Andrew Zhao and Yang Yue and Shiji Song and Bowen Yu and Gao Huang and Junyang Lin , booktitle=. Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement...

    2025

  43. [43]

    2023 , cdate=

    Woosuk Kwon and Zhuohan Li and Siyuan Zhuang and Ying Sheng and Lianmin Zheng and Cody Hao Yu and Joseph Gonzalez and Hao Zhang and Ion Stoica , title=. 2023 , cdate=

    2023

  44. [44]

    arXiv preprint arXiv:1909.08053 , year=

    Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism , author=. arXiv preprint arXiv:1909.08053 , year=

    Pith/arXiv arXiv 1909

  45. [45]

    arXiv preprint arXiv:2504.14945 , year=

    Learning to reason under off-policy guidance , author=. arXiv preprint arXiv:2504.14945 , year=

    Pith/arXiv arXiv

  46. [46]

    arXiv preprint arXiv:2506.07527 , year=

    Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questions , author=. arXiv preprint arXiv:2506.07527 , year=

    arXiv

  47. [47]

    arXiv preprint arXiv:2506.19767 , year=

    SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoning , author=. arXiv preprint arXiv:2506.19767 , year=

    Pith/arXiv arXiv

  48. [48]

    arXiv preprint arXiv:2509.04419 , year=

    Towards a unified view of large language model post-training , author=. arXiv preprint arXiv:2509.04419 , year=

    arXiv

  49. [49]

    arXiv preprint arXiv:2506.09340 , year=

    RePO: Replay-Enhanced Policy Optimization , author=. arXiv preprint arXiv:2506.09340 , year=

    Pith/arXiv arXiv

  50. [50]

    arXiv preprint arXiv:2507.06892 , year=

    Squeeze the soaked sponge: Efficient off-policy reinforcement finetuning for large language model , author=. arXiv preprint arXiv:2507.06892 , year=

    Pith/arXiv arXiv

  51. [51]

    arXiv preprint arXiv:2510.02245 , year=

    ExGRPO: Learning to reason from experience , author=. arXiv preprint arXiv:2510.02245 , year=

    arXiv

  52. [52]

    arXiv preprint arXiv:2507.07451 , year=

    Rlep: Reinforcement learning with experience replay for llm reasoning , author=. arXiv preprint arXiv:2507.07451 , year=

    Pith/arXiv arXiv

  53. [53]

    arXiv preprint arXiv:2510.04140 , year=

    Selective Expert Guidance for Effective and Diverse Exploration in Reinforcement Learning of LLMs , author=. arXiv preprint arXiv:2510.04140 , year=

    Pith/arXiv arXiv

  54. [54]

    arXiv preprint arXiv:2510.03865 , year=

    Unlocking reasoning capabilities in llms via reinforcement learning exploration , author=. arXiv preprint arXiv:2510.03865 , year=

    arXiv

  55. [55]

    arXiv preprint arXiv:2509.07430 , year=

    The choice of divergence: A neglected key to mitigating diversity collapse in reinforcement learning with verifiable reward , author=. arXiv preprint arXiv:2509.07430 , year=

    arXiv

  56. [56]

    The Twelfth International Conference on Learning Representations , year=

    On-policy distillation of language models: Learning from self-generated mistakes , author=. The Twelfth International Conference on Learning Representations , year=

  57. [57]

    arXiv preprint arXiv:2601.02780 , year=

    MiMo-V2-Flash Technical Report , author=. arXiv preprint arXiv:2601.02780 , year=

    Pith/arXiv arXiv

  58. [58]

    arXiv preprint arXiv:2603.22117 , year=

    On the direction of rlvr updates for llm reasoning: Identification and exploitation , author=. arXiv preprint arXiv:2603.22117 , year=

    arXiv

  59. [59]

    arXiv preprint arXiv:2603.19835 , year=

    Fipo: Eliciting deep reasoning with future-kl influenced policy optimization , author=. arXiv preprint arXiv:2603.19835 , year=

    arXiv

  60. [60]

    arXiv preprint arXiv:2603.22446 , year=

    Sparse but critical: A token-level analysis of distributional shifts in rlvr fine-tuning of llms , author=. arXiv preprint arXiv:2603.22446 , year=

    arXiv

  61. [61]

    arXiv preprint arXiv:2606.30420 , year=

    Experience Augmented Policy Optimization for LLM Reasoning , author=. arXiv preprint arXiv:2606.30420 , year=

    Pith/arXiv arXiv

  62. [62]

    arXiv preprint arXiv:2605.22156 , year=

    One-Way Policy Optimization for Self-Evolving LLMs , author=. arXiv preprint arXiv:2605.22156 , year=

    Pith/arXiv arXiv

  63. [63]

    arXiv preprint arXiv:2605.22703 , year=

    Clipping Bottleneck: Stabilizing RLVR via Stochastic Recovery of Near-Boundary Signals , author=. arXiv preprint arXiv:2605.22703 , year=

    Pith/arXiv arXiv