{"id":"0c028c82-8593-4fbf-8285-ef3d14a49170","arxiv_id":"2607.10481","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"ARMOR adds correct reference-policy anchor samples to each RL batch and optimizes a reference-mixture importance ratio, preventing validation collapse and extending performance gains.","lead":"This paper proposes a reinforcement-learning trick that stabilizes long training runs for math-reasoning language models: each batch gets one correct answer from an earlier checkpoint, and the training objective is adjusted with a reference-mixture ratio. The authors report that this avoids the usual late-training drop in validation accuracy and keeps improving on AIME and AMC benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ARMOR's gains may come from a query filter, not anchor stabilization: Algo. 1 drops every question the reference cannot solve in 4 samples, changing the training distribution relative to DAPO; no coverage statistics are reported.","rationale":"The reader's weakest_assumption focuses on whether the reference samples are generalizable solution patterns. My concern is adjacent but more concrete and arguably more load-bearing: the algorithm's data-collection procedure silently filters the training distribution by reference solvability. This is a direct confound for the central empirical claim that ARMOR 'effectively mitigates validation collapse' and 'unlocks sustained performance gains.' The paper's own ablations conflate the anchor injection with the query filter, so even if the mechanism were exactly as claimed, the current experiments cannot rule out that simply dropping hard-to-reference-solve questions explains the stability. Because this is an unaddressed selection effect rather than a demonstrated error, the appropriate disposition remains conditional: the central claim is plausible and the algebra of Eq. (6) is internally consistent, but the headline empirical result needs a control experiment that separates the filter from the proposed components. I do not recommend changing the reader's CONDITIONAL verdict; the concern strengthens the reasons for requiring that additional evidence.","tokens_in":18416,"tokens_out":14801,"duration_ms":172261,"concrete_test":"Add a DAPO + Filter control: run DAPO with the identical query acceptance rule from Algorithm 1 (Var(R_on|x)>0 and max(R_off|x)=1 using 4 reference samples), but do not add y_anc to the group and do not use the mixed IS ratio. Log the fraction and identities of accepted queries per step. Run on Qwen2.5-Math-7B for the same continual-training budget as the ARMOR run. If DAPO+Filter reproduces the ARMOR curve, the reported gains are due to query filtering rather than anchor stabilization; if it still collapses like DAPO, the filter is not the driver. A secondary analysis should compare accepted versus rejected queries on a difficulty proxy (e.g., base-model pass rate) to quantify the selection bias.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The empirical comparison is confounded by a query-level filter introduced in Algorithm 1, Line 13. A training query enters the batch only if, in addition to DAPO's mixed-outcome condition, at least one of the four reference-policy rollouts is correct (max(R_off|x)=1). Because the reference is initialized from the best checkpoint of the same training run, queries that the reference cannot solve within the 4-sample budget are silently excluded. The paper never reports the fraction of queries excluded by this condition, nor how exclusion correlates with question difficulty. If the filter removes a substantial number of hard queries, ARMOR may appear to stabilize training simply by training on an easier subset of questions, not because of anchor injection or the mixed IS ratio. The ablations in Fig. 3 do not isolate this filter: every variant with Anchor Rollout also applies the filter, and every variant without Anchor Rollout removes both the anchor and the filter. Thus the reported gains over DAPO cannot be causally attributed to the proposed mechanism without controlling for this selection effect.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies the over-optimization failure mode in reinforcement learning with verifiable rewards (RLVR) for LLM reasoning and proposes ARMOR. Section 3 presents an empirical study in a continual-training setup (DAPO on Qwen2.5-Math-7B, resumed from the best AIME24 checkpoint): reverse-KL penalties at β ≤ 0.1 delay but do not prevent validation collapse, while β = 1.0 stabilizes only by inducing stagnation, and a forward/reverse KL asymmetry is used to explain this. The authors conclude that passive KL regularization is structurally insufficient. ARMOR replaces the KL penalty with two components: (1) Anchor Rollout, which injects one reward-correct sample from a reference policy πref (initialized as the run's best checkpoint) into each response group via rejection sampling over four rollouts (Algorithm 1); and (2) Mixed Optimization, which uses the mixture IS ratio r_mix = (απθ + (1−α)πref)/(απθ_old + (1−α)πref), interpreted as an adaptive trust region (Eq. 6) whose gains on the mixture target are claimed to transfer to πθ (Sec. 4.2). Experiments on Qwen2.5-Math-7B and Qwen3-8B-Base with DAPO and QAE report gains on AIME24/25 and AMC while maintaining general-task performance (Table 1), supported by ablations of the two components, reference-reset frequency, and initialization quality. The central claim is that ARMOR prevents validation collapse and lifts the performance ceiling over extended training horizons.","tokens_in":18701,"tokens_out":23590,"duration_ms":258377,"significance":"If the empirical identification were clean, this would be a solid and practical contribution. The reverse-KL insufficiency analysis (Sec. 3) is informative, including the k2/k3 estimator checks and the forward-vs-reverse KL asymmetry; the algebra in Eq. (6) is correct; and the 'strictly translates' claim in Sec. 4.2 is actually the exact linear relation Δmix = αΔθ, which strengthens the argument. The paper ships its code, evaluates on external benchmarks, and its ablations (Figs. 3-4) directly probe the two proposed components; the Limitations section honestly acknowledges the added hyperparameters and the lack of convergence guarantees. However, the empirical core is currently confounded: Algorithm 1 introduces a query-level selection gate that no ablation isolates, and the headline results are single runs with checkpoints selected on the reported AIME24 metric. The contribution is therefore conditional on closing this identification gap.","major_comments":[{"comment":"The batch condition adds a query-level gate beyond DAPO's dynamic-sampling rule: a query is kept only if max(R_off|x)=1, i.e., the reference policy obtains at least one correct response within the 4-sample budget. Because πref is the run's own best checkpoint, this silently removes questions the current best model cannot solve, biasing the training distribution. The manuscript gives no statistics on the dropped-query fraction or its correlation with difficulty. Figs. 3a/3b compound this: every variant that includes anchors also includes the gate, and every variant without anchors removes both. Hence the reported gains over DAPO cannot be causally attributed to anchor injection or the mixed ratio until a control (e.g., DAPO with the identical gate and standard IS ratio but no anchor) and the gate statistics are provided.","section":"Algo. 1, Line 13 (Sec. 4.1, 5.2)"},{"comment":"All reported checkpoints are selected on the AIME24 validation metric, so the headline numbers are post-hoc maxima over the training trajectory, and each is a single run without error bars. The QAE re-run in App. C.3 (Fig. 6) shows run-to-run spread that cannot be assessed for the primary DAPO comparisons; the +11.15 gain on Qwen3 is one trajectory. The phrase 'significantly higher performance ceiling' (Sec. 4.2) and the abstract's claims require at least 2-3 seeds with mean±std or per-run curves for the headline configurations, and ideally metrics at matched step counts in addition to metric-selected checkpoints.","section":"Table 1, Sec. 5.1, Fig. 6"},{"comment":"The mixture model treats the anchor as drawn from πref, but Eq. (3) specifies y_anc ~ πref(·|x) conditioned on R=1, with rejection sampling and argmax over up to 4 attempts; πmix in Eq. (4) ignores this conditioning, so r_mix in Eq. (5) is not the exact likelihood ratio for anchor tokens. Also, the group is |G|−1 on-policy responses plus one anchor (App. B keeps |G|=16), so the natural α for Qwen2.5 is 15/16, which is used, but Qwen3 uses α=0.875, inconsistent with that composition. Please correct the ratio for the conditional anchor distribution or state explicitly that Eq. (5) is a heuristic approximation, and quantify the difference relative to the standard off-policy IS variant in Fig. 3b.","section":"Sec. 4.2, Eq. (5) vs. Eq. (3), App. B"},{"comment":"The mechanism's premise is that injected samples are 'established generalizable solution patterns.' Since πref is initialized from the best checkpoint of the same DAPO run (Sec. 3 footnote; Sec. 4.1), it may contain exactly the over-optimized modes that cause validation collapse, and rejection sampling on the training reward cannot separate generalizable from over-optimized correct solutions. Fig. 3c varies only the checkpoint used for initialization; it does not test anchor content. A diagnostic (e.g., validation accuracy of the anchor solutions, or comparison of anchors from the collapsed vs. stable checkpoint) is needed, because this assumption is load-bearing for the 'active mode retention' story.","section":"Sec. 4.1, Eq. (3), Fig. 3c"}],"minor_comments":[{"comment":"The citation 'Liu et al., 2025a' is ambiguous: the bibliography contains two distinct Liu et al. 2025a entries (Aixin Liu et al., DeepSeek-V3.2, and Jiacai Liu et al.). Please disambiguate.","section":"Footnote 3"},{"comment":"The 'strictly translates' implication is in fact an equivalence: with πref fixed within a step, Δmix = αΔθ exactly. State this directly; it strengthens rather than weakens the argument.","section":"Sec. 4.2"},{"comment":"The group-size arithmetic should be clarified in one place: Eq. (3) implies 15 on-policy + 1 anchor = 16 responses, which matches α=0.9375 for Qwen2.5 but not α=0.875 for Qwen3. Specify whether the anchor replaces or appends to the on-policy set, and how α is chosen for each model.","section":"App. B"},{"comment":"The 'Step' column semantics are unclear: for instance '+200' presumably means 200 steps after the best checkpoint (i.e., step 340 overall). Please state this explicitly so training budgets are comparable.","section":"Table 1"},{"comment":"The differentiation from the concurrent experience-guided preprints (EAPO, OWPO, FIPO) is purely conceptual. Given the shared authorship cluster, add at least a brief algorithmic comparison or a quantitative side-by-side result to substantiate the claimed difference from 'bootstrap a new model.'","section":"App. A"},{"comment":"The query-selection gate in Algo. 1, Line 13 is not discussed in the Limitations section. Given its confounding potential, it should be explicitly acknowledged and analyzed there.","section":"Limitations"},{"comment":"Many reference entries truncate author lists to 'and 1 others'; please use the journal's full-author formatting.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"My assessment aligns with the reader's: the load-bearing issue is the unidentified query filter in Algorithm 1, and the paper's central empirical claim is not identifiable until that is controlled. I would treat the filter control and seed-variance reporting as acceptance-blocking. Separately, I flag a novelty-disclosure concern: the reference list contains a dense cluster of overlapping preprints from the same group (EAPO, OWPO, FIPO, and related work on clipping/off-policy stabilization) that are distinguished only conceptually in App. A and never compared quantitatively. At least one of these (EAPO, 'Experience Augmented Policy Optimization') sounds close to the anchor-rollout idea. I am not alleging impropriety, but I suggest the editor request a precise algorithmic-delta statement relative to the most similar of these methods, and ideally a comparison."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, the core idea is genuinely useful: instead of a passive reverse-KL penalty, ARMOR injects a correct reference-policy sample into every response group and then optimizes a mixed IS ratio for the implied mixture policy. The trust-region boundary derivation in Eq. 6 is neat and the ablation against static clip expansion is a good check that the benefit isn't just a wider clip range. Second, the empirical section has a real confound that the paper never addresses. In Algorithm 1, line 13, a query only enters the batch if, in addition to DAPO's mixed-outcome condition, at least one of the four reference rollouts is correct. That means queries the reference can't solve in four samples are silently dropped. The paper reports no fraction of excluded queries and no coverage statistics, and the ablations don't isolate the filter: every anchor-rollout variant includes it, every no-anchor variant removes it. So the reported gains over DAPO could be partly from training on an easier, reference-solvable subset rather than from the anchor stabilization itself. That's not a fatal flaw on its own, but it is load-bearing for the causal claim, and it's fixable: report the exclusion rate, compare on the subset of queries that both methods see, and maybe add a control that applies the same filter without the anchor. Other soft spots are more minor: the main tables are single runs with checkpoints selected on the same AIME24 metric used as the headline, and the per-model choices of alpha and the reset schedule are manually tuned. The paper does include a QAE re-run and several reset-schedule variants, which helps, but formal error bars would be better. The KL-failure analysis in Section 3 is solid and useful on its own, and the related-work overlap with EAPO and other overlapping-author preprints is worth a clarifying sentence but isn't a disqualifier. Theory is light, as the authors acknowledge in the limitations. Who gets value: anyone working on RLVR stabilization or on off-policy guidance for reasoning models. It deserves a serious referee, but the reviewer should ask for the filter-control experiments and a held-out checkpoint-selection protocol before accepting the headline claim.","headline":"ARMOR is a plausible fix for late-training validation collapse, with a cute trust-region derivation, but the main empirical comparison is confounded by a silent query filter in the rollout algorithm and needs better controls before the headline numbers are credible.","tokens_in":19177,"tokens_out":1654,"would_cite":true,"duration_ms":21666,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ARMOR claims that injecting correct reference-policy samples and optimizing a mixed policy prevents the validation collapse that standard KL regularization cannot stop in long-horizon LLM reasoning RL.","keywords":["reinforcement learning","LLM reasoning","over-optimization","reverse KL regularization","validation collapse","anchor rollout","mixed policy optimization","adaptive trust region"],"falsifier":"Use ARMOR from a checkpoint whose correct-answer samples are known to be over-optimized shortcuts (a model trained past the validation peak), and compare validation trajectories against ARMOR from a clean peak checkpoint. If the anchor-sourced correct samples do not restore generalization, then keeping the reference's correct modes is not the operative mechanism; alternatively, run ARMOR with anchors sampled without the correctness filter and see whether validation collapse returns.","tokens_in":18344,"feed_emoji":"⚓","tokens_out":4866,"duration_ms":50135,"temperature":0.7,"pith_summary":"Long-horizon reinforcement learning for LLM reasoning can show steadily rising training reward while validation accuracy falls—a failure the paper calls over-optimization. The paper argues the standard defense, reverse KL regularization, is structurally insufficient: it is mode-seeking, so the policy can collapse onto a narrow set of high-reward patterns without penalty, and it applies a uniform penalty that, if strengthened, stalls exploration. ARMOR replaces the passive penalty with active stabilization: each training batch includes a correct sample generated by the reference policy (Anchor Rollout), and the policy objective is rewritten to optimize a mixture alpha*pi_theta + (1-alpha)*pi_ref (Mixed Optimization), which yields an adaptive trust region. The paper reports that this combination prevents validation collapse and sustains or improves reasoning scores on math benchmarks beyond the baseline's collapse point. A sympathetic reader would take the central claim to be that preserving known-good solution patterns through sampling, rather than penalizing drift, is the missing ingredient for stable RL scaling.","feed_headline":"Correct anchor samples stop LLM RL validation collapse","feed_subtitle":"Injecting known-good reference-policy answers and optimizing the mixed policy keeps reasoning gains climbing where KL penalties stall.","key_machinery":"The load-bearing object is the mixed importance-sampling ratio r_mix = (alpha*pi_theta + (1-alpha)*pi_ref) / (alpha*pi_theta_old + (1-alpha)*pi_ref), which makes the data distribution generated by Anchor Rollout—a mixture of on-policy samples and one correct anchor sample per query—consistent with the optimization target. Solving r_mix = 1 +/- epsilon gives the effective trust-region boundary pi_theta = (1 +/- epsilon)*pi_theta_old +/- epsilon*(1-alpha)/alpha*pi_ref; the second term is the adaptive expansion that lets updates be larger where the reference supports the action. The anchor sample itself is produced by rejection sampling from pi_ref with the requirement R(x, y_anc) = 1, so the r","core_discovery":"The central claim is that validation collapse in reinforcement learning with verifiable rewards is not primarily a reward-design problem but an algorithmic distributional one: the policy drifts off the reference distribution's generalizable modes even when rewards are verifiable. ARMOR counters this by injecting, for every query, one correct off-policy sample drawn from the reference policy via rejection sampling, ensuring the model keeps seeing established correct solutions; and by replacing the standard importance-sampling ratio with r_mix = (alpha*pi_theta + (1-alpha)*pi_ref) / (alpha*pi_theta_old + (1-alpha)*pi_ref). The paper proves that gains on the mixture target strictly transfer to","pith_inferences":["The same mechanism should transfer to coding or agentic tasks with verifiable rewards: if correct reference traces are injected and the objective is mixed, validation collapse from shortcut exploitation should be delayed there as well—a direct test of the paper's generality.","The rejection-sampling filter (only correct anchors) is probably what makes Anchor Rollout work; a natural control experiment would swap in unlabeled reference samples. If unlabeled anchors work equally well, the paper's explanation in terms of preserving correct modes would need revision.","The adaptive trust-region view suggests a parameter-free alternative: one could set alpha from the estimated overlap between pi_theta and pi_ref, so the expansion term grows exactly when the policy has drifted, instead of tuning alpha per model.","The paper's own limitation about computational overhead points to an offline replay buffer of high-quality anchors; if such a buffer preserves the benefit, the method becomes substantially cheaper and more practical."],"forward_implications":["Standard reverse KL regularization cannot bridge the validation gap: with small coefficients it only delays collapse, and with a large coefficient it freezes the policy; this holds for both k2 and k3 KL estimators.","Because gains on the mixture objective strictly imply gains on pi_theta, optimizing the mixed policy is a valid way to improve the target policy rather than a proxy that can diverge from it.","The clipping boundary expands where pi_ref assigns probability, so correct reference-supported tokens can receive larger positive updates and reference-favored wrong tokens stronger negative updates, without a uniform exploration penalty.","Periodic reset of the reference policy to the current policy prevents the anchor from becoming a fixed bottleneck; a static anchor stabilizes but caps the ceiling.","ARMOR can rescue a model already in a collapsed state, but starting from the best pre-collapse checkpoint yields a higher final ceiling."],"fun_headline_variants":["Anchor samples fix LLM RL instability","Stabilize LLM RL by mixing in reference answers","Off-policy anchors prevent RL reasoning collapse","ARMOR: off-policy anchors stabilize LLM RL"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The method assumes the correct samples drawn from the reference checkpoint are genuinely generalizable solution patterns; if the reference has already over-optimized onto shortcut modes, the anchor will preserve exactly those modes, and the stabilization mechanism would hold the model at the wrong answer.","fun_headline_variants_meta":{"raw":{"variants":["Anchor samples fix LLM RL instability","Stabilize LLM RL by mixing in reference answers","Off-policy anchors prevent RL reasoning collapse","ARMOR: off-policy anchors stabilize LLM RL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000186,"raw_usage":{"total_tokens":1141,"prompt_tokens":704,"completion_tokens":437,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":448,"completion_tokens_details":{"reasoning_tokens":389}},"tokens_in":448,"tokens_out":437,"duration_ms":4515,"temperature":1.0,"reasoning_tokens":389,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T07:13:29.236714+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Use ARMOR from a checkpoint whose correct-answer samples are known to be over-optimized shortcuts (a model trained past the validation peak), and compare validation trajectories against ARMOR from a clean peak checkpoint. If the anchor-sourced correct samples do not restore generalization, then keeping the reference's correct modes is not the operative mechanism; alternatively, run ARMOR with anchors sampled without the correctness filter and see whether validation collapse returns.","supporting_citations":[],"review_version":2}