{"id":"9879b2c0-8972-4716-8fa7-98ce6775447d","arxiv_id":"2608.03119","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Outcome-masked GRPO, which blocks gradients on answer tokens, prevents answer-collapse in label-free RLVR and performs on par with ground-truth-reward training.","lead":"The paper introduces a training rule that stops a language model from gaming label-free reinforcement learning by copying popular answers. The method masks answer tokens during gradient updates and adds cheap pairwise comparisons, producing stable training that roughly matches training with manually verified answers.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim depends on group-answer consensus being a reliable correctness proxy; if a confident wrong consensus forms, masking only slows drift and OM-GRPO could stably converge to the wrong answer instead of matching GT-reward.","rationale":"Read in good faith, the paper identifies a real failure mode, answer-token shortcut learning, and supports the masking mechanism with substantial experiments: Table 1, ablations in Table 3, stability curves, TTRL, weak- and strong-initialization analyses, and CAR's mechanistic checks. The central claim, however, has two coupled premises: (i) answer-frequency rewards are a valid correctness proxy; (ii) masking forces improvements to flow through reasoning. The paper proves a version of (ii) as a stability statement in Appendix D.2, but not (i). Theorem 2 is derived for hard majority-voting rewards, whereas the implemented objective uses the soft frequency reward of Eq. (2); even for MV rewards the theorem shows only that collapse is redirected to reasoning, not that the consensus answer is correct. A biased consensus can be amplified through reasoning tokens as slowly and stably as a correct one, and CAR can sharpen that bias. The reader's weakest_assumption points to exactly this gap, and I agree. This is not a claim that the paper is wrong on its benchmarks, since the evidence is strong, but the headline is stated as a general property, and the stability mechanism is orthogonal to reward correctness. Other flagged issues are real but secondary: the checkpoint-selection protocol is conservative for collapse-prone baselines and does not inflate OM-GRPO's advantage, and the KL-mask discrepancy between Eq. (1) and Appendix D.2 affects theory-implementation alignment rather than the core proxy assumption. The proposed wrong-modal subset test directly targets the missing link and would settle whether the concern lands; until then the CONDITIONAL verdict is appropriate.","tokens_in":27995,"tokens_out":10419,"duration_ms":110961,"concrete_test":"Using the Qwen3-1.7B-Base setup, compute each training prompt's modal answer from 8 rollouts before training; select the 1,000 prompts whose modal answer is wrong (the risky regime), plus a matched control with correct modal answers. Train OM-GRPO and GT-Reward from the same checkpoint on each subset with identical hyperparameters over the full 6-epoch budget, then compare final accuracy and answer diversity. Additionally, at checkpoints 50/150/300, report the per-group correlation between the soft reward of Eq. (2) and ground-truth correctness. If OM-GRPO tracks GT-Reward on the wrong-modal subset, the concern is resolved; if it stably converges to the wrong majority, the central 'matches GT-reward' claim fails precisely when consensus is unreliable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"OM-GRPO's headline claim requires that the answer-frequency reward in Eq. (2) is a reliable correctness proxy and that the masked update in Eq. (1) turns that alignment into better reasoning. Appendix D.2 proves only the stability half: under majority voting, masking eliminates the answer-token shortcut and redirects credit to reasoning (theta -> y -> h_T -> z), with KL making reasoning collapse costly. It does not prove that the consensus is correct. Eq. (2) is a pure frequency signal; a common wrong answer receives the same soft reward as a correct one. If the base model is biased or the distribution shifts so the group consensus is wrong, the masked gradient will reinforce reasoning that produces that wrong answer. The method can then be stable without being accurate: slow, non-collapsing drift toward the wrong consensus satisfies the stability claim while violating the 'matches GT-reward' claim. CAR (Section 3.3) amplifies whatever answer the policy reliably reproduces in pairwise comparisons and does not correct a biased consensus. The empirical defenses (weak-model tests, ReCEval, Qwen3-14B, low-contamination benchmarks) demonstrate that consensus was often reliable on these distributions, not that masking makes it reliable. The paper's Limitations section concedes it 'still relies on outcome-based reward signals rather than explicit supervision of reasoning quality.'","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes OM-GRPO, a label-free RLVR method that decouples reward estimation from policy optimization by (i) computing a soft frequency-based consensus reward over a group of sampled trajectories, (ii) masking the gradient and KL updates on the answer span, and (iii) augmenting the answer pool with cheap pairwise comparisons (CAR). The central claim is that this design prevents answer-token reward hacking and training collapse, outperforms existing label-free RLVR baselines, and matches ground-truth-reward RLVR, with a test-time-training gain of 4.24 points over majority voting. Evidence includes three backbones, nine benchmarks, component ablations, training-dynamics curves, weak-initialization and low-contamination tests, and a theoretical appendix on collapse prevention.","tokens_in":28288,"tokens_out":5397,"duration_ms":57881,"significance":"If the central claim holds, OM-GRPO is a simple and valuable contribution: it shows that masking answer-span gradients is sufficient to stop the known collapse in voting-based self-reward, without requiring gold labels or an extra reward model. The empirical effort is a genuine strength: three backbones, nine benchmarks, TTRL experiments, ablations separating OMU, soft reward, and CAR, plus explicit training-dynamics, answer-diversity, and consensus-accuracy analyses. The paper also reports wall-clock overhead transparently. The main risk is that the method inherits its correctness signal from group consensus, so the paper establishes stability more firmly than it establishes correctness; this is acknowledged in the Limitations section but should be framed more precisely in the main text and theory.","major_comments":[{"comment":"The baseline evaluation is asymmetric. Section 4.1 states that baselines with late-stage collapse are reported at their best validation checkpoint, but no equivalent statement is made for OM-GRPO or for GT-Reward. Table 1 therefore appears to compare OM-GRPO's final (or unspecified) checkpoint against baselines' best checkpoints, which can distort the claimed margin over label-free baselines and the comparison to GT-Reward. Please state the checkpoint-selection rule for every method and report both best and final checkpoints for OM-GRPO, or compare all methods under a single protocol.","section":"Section 4.1 and Table 1"},{"comment":"The main-text objective in Eq. (1) applies the token mask mi,t to the policy-gradient/clip term but not to the KL regularizer, whereas the theoretical objective in Appendix D.2.2, Eq. (20), masks both the policy-gradient term and the KL term on the answer span. Lemma 7 and Theorem 2 rely on the masked KL term for the \"KL barrier against reasoning collapse\" argument. As written, the theory does not apply to the stated main objective. Please align the equations or clarify which masking is actually used in the implementation, and adjust the proof accordingly.","section":"Eq. (1) and Eq. (20)"},{"comment":"Theorem 2 proves that masking answer-span gradients removes the direct answer-token shortcut and that reasoning collapse is KL-costly, but it does not prove that the group consensus is correct. The reward in Eq. (2) is a pure answer-frequency signal, so a consistently wrong consensus can receive the same high soft reward as a correct consensus; in that regime the masked objective would stably reinforce reasoning that produces the wrong answer. The empirical consensus-accuracy curves in Figure 6 and the weak-initialization and low-contamination experiments mitigate this concern for the tested distributions, and the Limitations section concedes the reliance on outcome-based signals. However, the main text phrase \"provably eliminated\" (Section 3.2) and the wording of Theorem 2 overstate the theoretical reach. Please state the correctness assumption explicitly, soften the proof claims accordingly, or add a formal treatment of the wrong-consensus case.","section":"Appendix D.2 and Section 3.1"},{"comment":"CAR uses the policy being trained as its own pairwise judge: the augmented answer pool is generated by the same model whose weights are being updated. The mechanistic analysis in Appendix E.7 shows that position bias decreases and that correct trajectories produce lower pairwise entropy, which is useful evidence. However, it does not include a control against an independent judge or a fixed reference policy over the full training run, so it remains possible that the comparison branch itself drifts with the policy. Please add such a control or clearly state that the pairwise comparisons are assumed to remain faithful throughout training.","section":"Section 3.3 and Appendix E.7"},{"comment":"The term \"avg@k setting\" is used for the main result, but the metric is not defined in the main text; it is only inferred from Appendix A that each benchmark uses a different k (e.g., avg@16 for AIME, avg@4 for MATH500 and GSM8K, avg@8 for AMC). Please define the aggregate metric in Section 4.2 and clarify whether the reported averages in Table 1 are unweighted means across benchmarks with different k values.","section":"Section 4.2 and footnote 3"}],"minor_comments":[{"comment":"In the second paragraph of Section 1, \"Majority V oting\" contains a stray space; this also appears in the baseline list in Section 4.1.","section":"Introduction"},{"comment":"The appendix title uses \"Overcome-Masked Update\" (in D.2) while the main text consistently uses \"Outcome-Masked Update\"; please unify the terminology.","section":"Appendix D"},{"comment":"The Table 5 caption reads \"Pass@k Results (%) ofRL performancecomparison\" and should be spaced and reworded for readability.","section":"Appendix E.2"},{"comment":"The caption and the in-text discussion in Appendix E.7 refer to panels (a)-(d), but the figure layout and the order of discussion make it hard to map the panels; please align the panel labels with the textual references.","section":"Figure 9"},{"comment":"The ablation paragraph reports average drops for \"w/o OMU\" on only two of the three backbones; for completeness, please report the corresponding drop for Llama-3.2-3B-Instruct as well.","section":"Section 4.4"}],"recommendation":"major_revision","confidential_remarks":"This is a solid empirical paper with unusually broad validation for a label-free RLVR method, and I believe the central idea is likely correct. My concerns are not about the value of the contribution but about whether the current text supports all of its claims: the checkpoint-selection asymmetry in Table 1, the mismatch between Eq. (1) and the theoretical objective in Eq. (20), and the need to frame the theory as a stability result rather than a correctness guarantee. These are addressable within the manuscript's scope, so I recommend major revision rather than rejection. The paper's fit with cs.AI is appropriate; no novelty-disclosure concerns arose from the references I checked."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on arXiv:2608.03119 (OM-GRPO). The core idea is clean and new as far as I know: in label-free RLVR, you compute the answer-level soft reward from group consensus, but you zero out the gradient on the answer-span tokens, forcing the policy to improve reasoning to earn reward. That directly targets the failure mode described—answer-token shortcut leading to collapse—and the paper demonstrates it with training-curve plots showing MV/CoReward collapse while OM-GRPO stays near GT-Reward.\n\nThe experimental effort is the strongest part. Three backbones, nine benchmarks, ablations that include removing each component, a TTRL setting, weak-base-model tests, low-contamination benchmarks, a ReCEval-style reasoning-quality check, and even PRM-based baselines that collapse. That's a lot of work and it mostly supports the claim. CAR, the contrast-augmented reward, is a sensible extension; the pairwise comparisons reuse existing trajectories and the cost analysis in Appendix F is honest.\n\nThe main soft spot is that the theory only proves the stability half. Appendix D.2 shows that masking removes the direct answer-token shortcut and that the KL barrier makes reasoning collapse expensive. It doesn't prove the soft frequency reward is a reliable correctness signal. The stress-test concern is valid: if the group consensus is confidently wrong, the method can stably converge to the wrong answer. The paper's own Limitations section concedes it still relies on outcome-based signals. So \"matches GT-reward\" is an empirical finding on these distributions, not a guaranteed property. The empirical defenses (weak models, low-contamination, ReCEval) help, but they don't eliminate the concern.\n\nTwo smaller issues. First, Section 4.2 gives collapsing baselines their best validation checkpoint; OM-GRPO's final checkpoint is reported. That asymmetry actually works against OM-GRPO, so it's not a favorable bias, but it should be stated and ideally both numbers reported. Second, Eq. (1) shows an unmasked KL term, while the Appendix D.2 objective masks it. The paper should say which KL is actually implemented, since the answer-token KL in Figure 5 is analyzed separately.\n\nNo code is released, which makes direct verification harder, but the method is simple enough to reimplement.\n\nOverall: a solid, within-subfield paper with a real mechanism and unusually broad evaluations. I'd send it to referees, expecting the KL/checkpoint points to be clarified. If I worked on label-free RLVR I'd cite it.","headline":"A genuinely new and simple fix—masking answer-span gradients—backed by broad experiments; the theory proves stability, not correctness, and the paper should reconcile Eq. (1) with the appendix and clarify checkpoint selection.","tokens_in":28777,"tokens_out":3593,"would_cite":true,"duration_ms":34540,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Label-free RLVR collapses when answer tokens are directly reinforced; masking answer-span gradients while keeping a soft answer-frequency reward makes it match ground-truth-reward training.","keywords":["label-free RLVR","GRPO","reward hacking","mode collapse","outcome masking","majority voting","contrast-augmented reward","test-time training"],"falsifier":"On a synthetic benchmark where the correct answer varies but the model's group majority is wrong on nearly every prompt (for example, all sampled traces converge on a common incorrect value), train OM-GRPO to convergence and check whether Pass@8 accuracy improves. If accuracy does not improve while answer diversity stays high, the frequency-based soft reward is not supplying a correctness signal and the claim that label-free training matches ground-truth training fails in that regime.","tokens_in":27816,"feed_emoji":"🎯","tokens_out":8467,"duration_ms":76405,"temperature":0.7,"pith_summary":"The paper tries to establish that reinforcement learning for language-model reasoning can be trained without ground-truth answers if the training signal is kept off the final answer tokens. Its claim is that voting-based label-free RLVR collapses because the same answer-consensus signal is used both to score trajectories and to reinforce answer tokens, so a policy can raise reward by sharpening popular answer strings instead of reasoning better. OM-GRPO keeps the answer-level reward but masks the answer span from gradient updates, forcing all learning through the reasoning chain; a soft frequency reward replaces hard majority voting, and Contrast-Augmented Reward adds cheap pairwise comparisons to make the frequency signal more reliable. Reported experiments across three backbones and math, code, and instruction benchmarks show stable training over ten epochs, performance comparable to ground-truth-reward training, and a 4.24-point gain over majority voting in test-time training. If the claim holds, RLVR loses its main annotation bottleneck.","feed_headline":"Answer masking stops label-free RLVR collapse","feed_subtitle":"Blocking gradients on answer spans lets consensus-only rewards train reasoning stably, matching ground-truth supervision.","key_machinery":"The central object is the outcome mask $m_{i,t}$ in the masked GRPO objective, a binary token-level mask set to 0 on the answer span $z_i$ and 1 on the reasoning chain $y_i$, applied to both the clipped policy-gradient term and the KL regularizer. Its job is to block the direct pathway $\\theta \\to z$ while leaving the pathway $\\theta \\to y \\to h_T \\to z$ open, so reward gains must come from better reasoning. The objective is driven by a soft reward $r_i^q = \\Pr(z = z_i \\mid z \\in \\mathcal{Z}_{\\mathrm{aug}})$, the answer's frequency share in the group answer pool, plus a format reward; advantage $\\hat{A}_i$ standardizes these within each group as in GRPO. Contrast-Augmented Reward (CAR) enlarges the pool from $G$ trajectories to $O(G^2)$ by prompting the model to answer the original question after seeing pairs of existing reasoning traces, using the extra answers only for reward estimation, not as training trajectories. Together the mask and the soft frequency reward enforce that the only surviving optimization path is improving the reasoning that supports an answer.","core_discovery":"Label-free RLVR does not have to be a fragile stand-in for gold-supervised training. The paper demonstrates that the collapse seen in majority-voting self-rewarding is driven by answer-token over-optimization: a globally shared answer bias receives positive feedback from every group in which it is the mode, and under token-wise KL regularization the cheapest way to satisfy the reward is to shift probability mass on the short answer span rather than across the long reasoning trace. OM-GRPO removes that shortcut by setting the gradient mask to zero on the answer span for both the policy-gradient and KL terms, while still computing answer-level rewards. The paper further shows that a soft reward proportional to answer frequency in the group provides an implicit contrastive signal, and that augmenting the answer pool with pairwise comparisons (CAR) improves the reliability of the consensus. With these pieces, the trained policy matches supervised ground-truth-reward training on average while keeping answer diversity high and the reasoning–answer link intact. The theoretical analysis proves that the masked objective eliminates direct answer-level reinforcement and makes reasoning collapse KL-expensive, so the attracting collapsed state of vanilla majority-voting GRPO is no longer reachable through the same shortcut.","pith_inferences":["If masking works by removing a short, low-KL shortcut, the same principle may transfer to other RLVR settings where a short output span carries the reward, such as tool calls, code answers, or structured outputs; this is an extension the paper does not test.","The theoretical argument shows the collapse mechanism is answer-frequency driven and independent of correctness, so OM-GRPO cannot by itself fix a distribution where the group consensus is systematically wrong; pairing it with an occasional external verifier would be a natural next test.","The paper's experiments cap at 7B open models, so whether the stability persists at larger scales is untested, as the paper itself notes.","CAR's pairwise comparisons are produced by the same policy being trained, creating a self-referential loop; comparing CAR against a fixed external judge or a frozen reference model could isolate whether the benefit comes from genuine comparison quality or from distributional self-consistency."],"forward_implications":["Label-free RLVR can be run for many epochs without the sharp accuracy collapse that majority-voting, confidence, entropy, and agreement-based baselines exhibit.","Without any gold answers, the method reaches average performance comparable to supervised ground-truth-reward training across MATH, GSM8K, AMC, AIME, LiveCodeBench, CRUX, IFEval, and MMLU-Pro.","Contrast-Augmented Reward gives better reward estimates than increasing the rollout budget, outperforming n=16 sampling at comparable cost and beating n=64 sampling by 3.64x in training time.","Outcome masking transfers to test-time training, where it prevents collapse and outperforms majority voting by 4.24 points, and to open-ended generation with entropy-based rewards.","Removing the mask reintroduces collapse, and partial answer-token weighting at 0.5 or 0.75 loses stability, so full masking is the reliable configuration."],"supporting_citations":[{"why":"Supplies the GRPO group-relative objective and advantage normalization that OM-GRPO modifies with its answer-span mask.","marker":"Shao et al., 2024"},{"why":"Provides self-consistency majority voting over sampled answers, the reward proxy that OM-GRPO keeps but softens.","marker":"Wang et al., 2023"},{"why":"Represents the voting-based self-rewarding baseline whose answer-token collapse the paper diagnoses and fixes.","marker":"Shafayat et al., 2025"},{"why":"CoReward is the strongest label-free baseline compared against, aggregating consensus over paraphrased inputs.","marker":"Zhang et al., 2025"},{"why":"Entropy-based label-free baseline; the paper shows outcome masking also stabilizes this reward family.","marker":"Prabhudesai et al., 2025"},{"why":"MATH supplies the training split and MATH5000 validation set used in the main experiments.","marker":"Hendrycks et al., 2021"}],"fun_headline_variants":["OM-GRPO masks answer gradients to fix RLVR collapse","Hiding answers from gradients stops label-free RLVR gaming","Answer-masked rewards match gold-supervised RLVR without labels","Gradient-masked answers stabilize label-free RLVR training"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that group answer-frequency in a soft reward is a reliable enough proxy for actual correctness; if the sampled group mostly agrees on a wrong answer, masking slows the drift toward that wrong answer but does not remove it.","fun_headline_variants_meta":{"raw":{"variants":["OM-GRPO masks answer gradients to fix RLVR collapse","Hiding answers from gradients stops label-free RLVR gaming","Answer-masked rewards match gold-supervised RLVR without labels","Gradient-masked answers stabilize label-free RLVR training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000206,"raw_usage":{"total_tokens":1416,"prompt_tokens":983,"completion_tokens":433,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":599,"completion_tokens_details":{"reasoning_tokens":364}},"tokens_in":599,"tokens_out":433,"duration_ms":4826,"temperature":1.0,"reasoning_tokens":364,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T00:51:34.024837+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a synthetic benchmark where the correct answer varies but the model's group majority is wrong on nearly every prompt (for example, all sampled traces converge on a common incorrect value), train OM-GRPO to convergence and check whether Pass@8 accuracy improves. If accuracy does not improve while answer diversity stays high, the frequency-based soft reward is not supplying a correctness signal and the claim that label-free training matches ground-truth training fails in that regime.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Represents the voting-based self-rewarding baseline whose answer-token collapse the paper diagnoses and fixes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"MATH supplies the training split and MATH5000 validation set used in the main experiments."}],"review_version":1}