ARMOR: Stabilizing On-Policy LLM RL with Off-Policy Anchor Samples

Chiyu Ma; Guoyin Wang; Jiancan Wu; Jinda Lu; Jingren Zhou; Junkang Wu; Kexin Huang; Shuo Yang; Xiangnan He; Xiang Wang

Active off-policy anchors and mixed policy ratios stop LLM RL from collapsing under over-optimization, where reverse KL fails.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 11:22 UTC pith:XE2G3UOS

load-bearing objection Solid practical fix for late-stage RLVR collapse; evidence is strongest for recovery after the gap appears, not yet for prevention from step 0. the 3 major comments →

arxiv 2607.10481 v2 pith:XE2G3UOS submitted 2026-07-11 cs.LG cs.AIcs.CL

ARMOR: Stabilizing On-Policy LLM RL with Off-Policy Anchor Samples

Kexin Huang , Junkang Wu , Jinda Lu , Shuo Yang , Chiyu Ma , Jiancan Wu , Xiang Wang , Xiangnan He

show 2 more authors

Guoyin Wang Jingren Zhou

This is my paper

classification cs.LG cs.AIcs.CL

keywords reinforcement learninglarge language modelsover-optimizationpolicy optimizationoff-policy samplesmathematical reasoningKL regularizationtrust region

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Reinforcement learning boosts large language model reasoning, yet late training often shows a reward-validation gap: training reward rises while held-out accuracy falls. The usual defense, reverse KL regularization, cannot simultaneously retain diverse correct solutions and permit exploration of better ones. This paper establishes that the failure is structural—mode-seeking reverse KL allows collapse onto shortcuts, while a heavy penalty freezes progress—and proposes replacing the passive penalty with active sample stabilization. By injecting one verified correct answer drawn from a reference policy into every training group and then optimizing a mixture of current and reference policies, the method keeps established solution patterns alive while still allowing controlled gains. Experiments on math reasoning benchmarks show that the approach turns collapsing runs into continued improvement over hundreds of extra steps, preserving or improving general capabilities as well. A reader cares because long-horizon RL for reasoning is currently brittle, and a practical way to keep scaling it would change how far these systems can be pushed.

Core claim

The paper claims that reverse KL is structurally insufficient against over-optimization in LLM reasoning RL because of its mode-seeking nature and uniform penalty, and that ARMOR—Anchor Rollout of correct reference samples plus Mixed Optimization of the mixture policy via a mixed importance-sampling ratio—mitigates validation collapse and enables sustained performance gains over extended training horizons.

What carries the argument

ARMOR: Anchor Rollout injects one correct off-policy sample from the reference into each response group; Mixed Optimization replaces the ordinary importance ratio with the mixture ratio of α·current + (1−α)·reference, producing an adaptive trust region that expands or contracts with the reference probability.

Load-bearing premise

A single correct answer drawn from the current best reference, mixed at a fixed coefficient and reset on a hand-tuned schedule, is enough to keep the generalizable solution modes of the reference alive.

What would settle it

Continue training from a peak checkpoint with ARMOR for several hundred steps on AIME24; if avg@32 still collapses while a carefully tuned reverse-KL baseline does not, the claim that active anchors outperform passive KL is falsified.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Training can continue past the usual collapse point with rising validation scores on AIME and AMC.
General capabilities on GPQA and MMLU-Pro are preserved or improved rather than traded away.
The same recipe works on top of both DAPO and QAE and across different base models.
Pass@k stays stable or rises, showing the gains are not merely accuracy-versus-coverage trade-offs.
Periodic reference resets are required; a static anchor eventually becomes a performance bottleneck.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Requiring only positive anchors implies the method is closer to selective imitation of verified trajectories than unrestricted exploration.
Active mode retention via a few correct samples may help other RL settings that suffer mode collapse without needing new divergence families.
Hybrid reset schedules that start aggressive then freeze may become standard for strong base models that need early plasticity.
If anchor sampling cost can be amortized with a replay buffer of high-quality correct answers, the overhead becomes low enough for routine use.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Solid practical fix for late-stage RLVR collapse; evidence is strongest for recovery after the gap appears, not yet for prevention from step 0.

read the letter

The useful takeaway is simple: reverse KL does not stop the reward–validation gap on math RL, and injecting one correct reference sample plus a mixed importance-sampling ratio does. That combination is the real contribution.

What is new is not off-policy data or mixture policies in the abstract—those already exist—but the concrete recipe: positive-only rejection-sampled anchors from the current best checkpoint, plus the mixed IS ratio that expands the trust region proportionally to π_ref. The KL diagnosis is clean (forward vs reverse asymmetry after 100 steps). Ablations separate the two pieces: Anchor Rollout alone stabilizes, Mixed Optimization alone raises the ceiling then collapses, and static clip expansion or plain off-policy IS do not match the full method. Results hold on two base models and two algorithms (DAPO, QAE), with pass@k and general-task numbers that do not look like pure trade-offs. Code is promised; the algorithm is fully specified.

Soft spots are real but secondary. All main tables and figures use a two-stage continual-training protocol: run until collapse, then resume from the peak (or even a collapsed) checkpoint with a hand-chosen reset schedule (τ=160 for 7B, hybrid τ=1 then freeze for 8B). Reset frequency and initial-anchor quality clearly move the final ceiling. So the paper shows reliable recovery and continued gains once over-optimization is already diagnosed; it does not yet show that the same fixed defaults would have prevented collapse from step 0. Alpha and the reference sampling budget are free parameters. Multi-seed variance is missing. These are the usual methods-paper gaps, not load-bearing contradictions.

This is for people who actually run long-horizon math/coding RL and keep hitting the validation cliff. It deserves a serious referee. I would engage with it, cite the recipe when I next hit the same failure mode, and ask for a from-scratch schedule and error bars in revision.

Referee Report

3 major / 4 minor

Summary. The paper argues that over-optimization in on-policy LLM RL for reasoning produces a reward–validation gap even under verifiable rewards, and that reverse KL regularization is structurally insufficient because of its mode-seeking character (high residual forward KL) and its uniform penalty (stagnation under large β). It proposes ARMOR, which replaces the passive KL term by (i) Anchor Rollout—injecting one correct off-policy sample drawn from a reference policy into every response group—and (ii) Mixed Optimization—replacing the ordinary importance-sampling ratio by the mixture ratio of Eq. (5) that corresponds to the hybrid data distribution. Continual-training experiments on Qwen2.5-Math-7B and Qwen3-8B-Base with DAPO and QAE show that ARMOR recovers or surpasses the pre-collapse peak on AIME/AMC while preserving or improving general-task scores, with ablations isolating the two components and the effect of reference-reset frequency.

Significance. If the claims hold, ARMOR supplies a practical, drop-in stabilizer for long-horizon RLVR that is orthogonal to system-level training–inference fixes and to reward-model improvements. The multi-model, multi-algorithm empirical base, the clean forward/reverse KL measurements, the component ablations (Figs. 3–4), the comparison against teacher-guided distillation baselines, and the public code release are genuine strengths that make the work immediately usable by the community. The adaptive-trust-region derivation (Eq. 6) also offers a clean conceptual link between mixture sampling and PPO-style clipping.

major comments (3)

All headline results (Table 1, Figs. 1, 3, 4) are obtained in a two-stage continual-training protocol that first runs the baseline until the reward–validation gap appears and then resumes from the peak (or even a collapsed) checkpoint with a model-specific reset schedule τ. Consequently the evidence primarily demonstrates recovery after over-optimization has already been diagnosed, rather than that the same mechanism would have prevented collapse if applied from step 0 with a fixed default schedule. A from-scratch comparison (or a clear statement that the method is intended only as a recovery tool) is needed to support the broader claim of a general stabilizer for long-horizon on-policy LLM RL.
Section 4.1 and Algorithm 1 assert that a single correct anchor sample per group is sufficient to retain the generalizable modes of the reference. The paper itself shows that negative anchors increase instability and that both α and the reset interval τ must be retuned per model (α = 0.9375 vs 0.875; τ = 160 vs hybrid τ = 1 then freeze). Without a sensitivity study that quantifies how much of the reported gain survives under a single fixed (α, τ) schedule, the claim that ARMOR is robust remains only partially supported.
The Mixed-Optimization derivation (Eqs. 4–6) equates improvement of the mixture policy with improvement of π_θ. The implication is stated only for the expected reward difference; it is not shown that the clipped surrogate used in practice (the DAPO/QAE objective with the mixed ratio) inherits the same monotonicity. A short formal argument or a counter-example would strengthen the theoretical grounding of the adaptive trust region.

minor comments (4)

Figure 2b reports KL values after only 100 steps; a longer-horizon plot would better illustrate whether the forward–reverse asymmetry persists or eventually closes under standard β.
The notation for the mixed IS ratio (Eq. 5) uses the same symbol rmix for both the ratio and the clipping argument; a distinct symbol for the clipped surrogate would improve readability.
Appendix C.3 shows that continuous reset (τ = 1) eventually degrades on Qwen3; the main text could briefly flag this failure mode so readers do not treat frequent resets as universally safe.
Table 1 reports a mild regression on GPQA/MMLU-Pro for the QAE+ARMOR setting; a one-sentence discussion of whether this is attributable to QAE’s advantage masking of the anchor would be helpful.

Circularity Check

0 steps flagged

No load-bearing circularity; algebraic consequences of mixture IS ratio and external-benchmark gains are independent of definitional loops or fitted-as-prediction steps.

full rationale

The paper's central derivation (Anchor Rollout + Mixed Optimization) does not reduce by construction to its inputs. Anchor Rollout is an explicit sampling procedure (Eq. 3, Algo. 1) that injects a rejection-sampled correct reference trajectory; Mixed Optimization simply substitutes the mixture probabilities into the IS ratio (Eq. 5) and algebraically expands the resulting clip boundary (Eq. 6). The implication that a positive mixture advantage yields a positive on-policy advantage follows immediately from linearity of expectation under a convex combination with fixed π_ref and α>0; it is a transparent algebraic property, not a self-definitional claim or a fitted quantity re-labeled as prediction. Empirical claims rest on held-out reasoning benchmarks (AIME24/25, AMC, GPQA, MMLU-Pro) whose labels are never used inside the training objective. Checkpoint selection on AIME24 introduces ordinary selection optimism but does not make the reported deltas tautological. Self-citations (chiefly the concurrent QAE work of overlapping authors) appear only as alternative baselines, not as uniqueness theorems or load-bearing premises that force the ARMOR construction. No ansatz is smuggled via citation, no uniqueness result is imported from prior author papers, and no parameter fitted on a subset is later called a first-principles prediction. The continual-training protocol and hand-tuned reset schedules affect the strength of the empirical claim but are outside the circularity criteria. Hence the derivation chain is self-contained; score remains near zero.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 2 invented entities

The central claim rests on standard RLVR machinery plus a small set of hand-chosen mixing and reset hyperparameters and the modeling choice that a single correct reference sample per group is enough to retain modes. No new physical entities are invented; the free parameters are the usual algorithmic knobs that must be set for the method to work as advertised.

free parameters (4)

mixing coefficient alpha = 0.9375 / 0.875 (model-specific)
Set to 0.9375 for Qwen2.5-Math-7B and 0.875 for Qwen3-8B-Base; controls the width of the adaptive trust region and the weight of the reference in the mixed IS ratio.
reference reset interval tau = 160 or hybrid schedule
Chosen per model (fixed 160 for 7B; hybrid tau=1 then freeze for 8B); determines how often the anchor is allowed to evolve.
reference sampling budget |Y_ref| = 4
Set to 4 per query for rejection sampling of a correct anchor; directly affects compute overhead and success rate of finding R=1 anchors.
clip ranges epsilon_low / epsilon_high = 0.2 / 0.28
Inherited from DAPO (0.2 / 0.28) but interact with the expanded mixed boundary; still free algorithmic choices.

axioms (4)

domain assumption Verifiable binary rewards (correct/incorrect) are consistent between training and validation, so observed collapse is algorithmic over-optimization rather than classic reward hacking.
Stated in Introduction and used to distinguish the failure mode from proxy-reward hacking.
ad hoc to paper A single correct off-policy sample per group is sufficient to retain the generalizable modes of the reference distribution.
Core design choice of Anchor Rollout (Eq. 3, Algo. 1); negative anchors are reported to increase instability.
domain assumption Performance gains on the mixture policy pi_mix strictly translate to gains on the target policy pi_theta.
Claimed after Eq. 5; used to justify optimizing the mixed IS ratio instead of an auxiliary KL loss.
standard math Standard PPO/GRPO/DAPO clipping and advantage estimation remain valid when the IS ratio is replaced by the mixed ratio.
Implicit in the Mixed Optimization derivation (Sec. 4.2–4.3).

invented entities (2)

Anchor sample y_anc no independent evidence
purpose: Explicit positive off-policy token sequence forced into every response group to prevent mode collapse.
Defined in Eq. 3 and Algo. 1; not a physical entity but a new algorithmic object whose necessity is argued empirically.
Mixed policy pi_mix = alpha pi_theta + (1-alpha) pi_ref no independent evidence
purpose: Optimization target that induces an adaptive trust region without an auxiliary KL term.
Introduced in Eq. 4–6; the adaptive boundary is derived, not independently measured outside the paper.

pith-pipeline@v1.1.0-grok45 · 23382 in / 3207 out tokens · 35410 ms · 2026-07-14T11:22:08.240344+00:00 · methodology

0 comments

read the original abstract

Reinforcement learning (RL) has significantly enhanced the reasoning capabilities of large language models (LLMs), yet the training process remains notoriously fragile. In this work, we investigate a critical source of this instability: over-optimization, where models exploit training heuristics at the expense of generalizable reasoning. While reverse KL regularization is the standard defense against such degradation, our analysis reveals that it is often insufficient in this regime, as it fails to ensure comprehensive coverage of the reference distribution. To address this, we propose ARMOR (Anchor Rollout and Mixed Optimization for RL), a framework that shifts the paradigm from passive penalty to active sample stabilization. ARMOR comprises two key components: (1) Anchor Rollout, which leverages off-policy data from the reference policy to preserve established solution patterns; and (2) Mixed Optimization, which reformulates the policy objective to enable controlled exploration without relying on auxiliary losses. Extensive experiments on reasoning benchmarks validate that ARMOR effectively mitigates validation collapse, enabling sustained performance improvements over extended training horizons.

Figures

Figures reproduced from arXiv: 2607.10481 by Chiyu Ma, Guoyin Wang, Jiancan Wu, Jinda Lu, Jingren Zhou, Junkang Wu, Kexin Huang, Shuo Yang, Xiangnan He, Xiang Wang.