Pith. sign in

REVIEW 5 major objections 5 minor 40 references

Don't Peek at the Answer: Outcome-Masked Group Relative Policy Optimization for Label-Free RLVR

T0 review · 5 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Label-free RLVR collapses when answer tokens are directly reinforced; masking answer-span gradients while keeping a soft answer-frequency reward makes it match ground-truth-reward training.

desk verdict A genuinely new and simple fix—masking answer-span gradients—backed by broad experiments; the theory proves stability, not correctness, and the paper should reconcile Eq. (1) with the appendix and clarify checkpoint selection. read the letter →

arxiv 2608.03119 v1 pith:UU7TUH2F submitted 2026-08-04 cs.AI

Yongshi Ye , Liang Zhang , Yidong Chen , Xiaodong Shi , Biao Fu This is my paper · ORCID
classification cs.AI
keywords label-freeRLVRGRPOrewardhackingmodecollapseoutcomemaskingmajorityvotingcontrast-augmentedtest-timetraining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that reinforcement learning for language-model reasoning can be trained without ground-truth answers if the training signal is kept off the final answer tokens. Its claim is that voting-based label-free RLVR collapses because the same answer-consensus signal is used both to score trajectories and to reinforce answer tokens, so a policy can raise reward by sharpening popular answer strings instead of reasoning better. OM-GRPO keeps the answer-level reward but masks the answer span from gradient updates, forcing all learning through the reasoning chain; a soft frequency reward replaces hard majority voting, and Contrast-Augmented Reward adds cheap pairwise comparisons to make the frequency signal more reliable. Reported experiments across three backbones and math, code, and instruction benchmarks show stable training over ten epochs, performance comparable to ground-truth-reward training, and a 4.24-point gain over majority voting in test-time training. If the claim holds, RLVR loses its main annotation bottleneck.

What carries the argument

The central object is the outcome mask $m_{i,t}$ in the masked GRPO objective, a binary token-level mask set to 0 on the answer span $z_i$ and 1 on the reasoning chain $y_i$, applied to both the clipped policy-gradient term and the KL regularizer. Its job is to block the direct pathway $\theta \to z$ while leaving the pathway $\theta \to y \to h_T \to z$ open, so reward gains must come from better reasoning. The objective is driven by a soft reward $r_i^q = \Pr(z = z_i \mid z \in \mathcal{Z}_{\mathrm{aug}})$, the answer's frequency share in the group answer pool, plus a format reward; advantage $\hat{A}_i$ standardizes these within each group as in GRPO. Contrast-Augmented Reward (CAR) enlarges the pool from $G$ trajectories to $O(G^2)$ by prompting the model to answer the original question after seeing pairs of existing reasoning traces, using the extra answers only for reward estimation, not as training trajectories. Together the mask and the soft frequency reward enforce that the only surviving optimization path is improving the reasoning that supports an answer.

What would settle it

On a synthetic benchmark where the correct answer varies but the model's group majority is wrong on nearly every prompt (for example, all sampled traces converge on a common incorrect value), train OM-GRPO to convergence and check whether Pass@8 accuracy improves. If accuracy does not improve while answer diversity stays high, the frequency-based soft reward is not supplying a correctness signal and the claim that label-free training matches ground-truth training fails in that regime.

Watch

Extended reading notes

Core claim

Label-free RLVR does not have to be a fragile stand-in for gold-supervised training. The paper demonstrates that the collapse seen in majority-voting self-rewarding is driven by answer-token over-optimization: a globally shared answer bias receives positive feedback from every group in which it is the mode, and under token-wise KL regularization the cheapest way to satisfy the reward is to shift probability mass on the short answer span rather than across the long reasoning trace. OM-GRPO removes that shortcut by setting the gradient mask to zero on the answer span for both the policy-gradient and KL terms, while still computing answer-level rewards. The paper further shows that a soft reward proportional to answer frequency in the group provides an implicit contrastive signal, and that augmenting the answer pool with pairwise comparisons (CAR) improves the reliability of the consensus. With these pieces, the trained policy matches supervised ground-truth-reward training on average while keeping answer diversity high and the reasoning–answer link intact. The theoretical analysis proves that the masked objective eliminates direct answer-level reinforcement and makes reasoning collapse KL-expensive, so the attracting collapsed state of vanilla majority-voting GRPO is no longer reachable through the same shortcut.

Load-bearing premise

The load-bearing premise is that group answer-frequency in a soft reward is a reliable enough proxy for actual correctness; if the sampled group mostly agrees on a wrong answer, masking slows the drift toward that wrong answer but does not remove it.

Editorial extensions

If this is right

  • Label-free RLVR can be run for many epochs without the sharp accuracy collapse that majority-voting, confidence, entropy, and agreement-based baselines exhibit.
  • Without any gold answers, the method reaches average performance comparable to supervised ground-truth-reward training across MATH, GSM8K, AMC, AIME, LiveCodeBench, CRUX, IFEval, and MMLU-Pro.
  • Contrast-Augmented Reward gives better reward estimates than increasing the rollout budget, outperforming n=16 sampling at comparable cost and beating n=64 sampling by 3.64x in training time.
  • Outcome masking transfers to test-time training, where it prevents collapse and outperforms majority voting by 4.24 points, and to open-ended generation with entropy-based rewards.
  • Removing the mask reintroduces collapse, and partial answer-token weighting at 0.5 or 0.75 loses stability, so full masking is the reliable configuration.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If masking works by removing a short, low-KL shortcut, the same principle may transfer to other RLVR settings where a short output span carries the reward, such as tool calls, code answers, or structured outputs; this is an extension the paper does not test.
  • The theoretical argument shows the collapse mechanism is answer-frequency driven and independent of correctness, so OM-GRPO cannot by itself fix a distribution where the group consensus is systematically wrong; pairing it with an occasional external verifier would be a natural next test.
  • The paper's experiments cap at 7B open models, so whether the stability persists at larger scales is untested, as the paper itself notes.
  • CAR's pairwise comparisons are produced by the same policy being trained, creating a self-referential loop; comparing CAR against a fixed external judge or a frozen reference model could isolate whether the benefit comes from genuine comparison quality or from distributional self-consistency.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes OM-GRPO, a label-free RLVR method that decouples reward estimation from policy optimization by (i) computing a soft frequency-based consensus reward over a group of sampled trajectories, (ii) masking the gradient and KL updates on the answer span, and (iii) augmenting the answer pool with cheap pairwise comparisons (CAR). The central claim is that this design prevents answer-token reward hacking and training collapse, outperforms existing label-free RLVR baselines, and matches ground-truth-reward RLVR, with a test-time-training gain of 4.24 points over majority voting. Evidence includes three backbones, nine benchmarks, component ablations, training-dynamics curves, weak-initialization and low-contamination tests, and a theoretical appendix on collapse prevention.

Significance. If the central claim holds, OM-GRPO is a simple and valuable contribution: it shows that masking answer-span gradients is sufficient to stop the known collapse in voting-based self-reward, without requiring gold labels or an extra reward model. The empirical effort is a genuine strength: three backbones, nine benchmarks, TTRL experiments, ablations separating OMU, soft reward, and CAR, plus explicit training-dynamics, answer-diversity, and consensus-accuracy analyses. The paper also reports wall-clock overhead transparently. The main risk is that the method inherits its correctness signal from group consensus, so the paper establishes stability more firmly than it establishes correctness; this is acknowledged in the Limitations section but should be framed more precisely in the main text and theory.

major comments (5)
  1. [Section 4.1 and Table 1] The baseline evaluation is asymmetric. Section 4.1 states that baselines with late-stage collapse are reported at their best validation checkpoint, but no equivalent statement is made for OM-GRPO or for GT-Reward. Table 1 therefore appears to compare OM-GRPO's final (or unspecified) checkpoint against baselines' best checkpoints, which can distort the claimed margin over label-free baselines and the comparison to GT-Reward. Please state the checkpoint-selection rule for every method and report both best and final checkpoints for OM-GRPO, or compare all methods under a single protocol.
  2. [Eq. (1) and Eq. (20)] The main-text objective in Eq. (1) applies the token mask mi,t to the policy-gradient/clip term but not to the KL regularizer, whereas the theoretical objective in Appendix D.2.2, Eq. (20), masks both the policy-gradient term and the KL term on the answer span. Lemma 7 and Theorem 2 rely on the masked KL term for the "KL barrier against reasoning collapse" argument. As written, the theory does not apply to the stated main objective. Please align the equations or clarify which masking is actually used in the implementation, and adjust the proof accordingly.
  3. [Appendix D.2 and Section 3.1] Theorem 2 proves that masking answer-span gradients removes the direct answer-token shortcut and that reasoning collapse is KL-costly, but it does not prove that the group consensus is correct. The reward in Eq. (2) is a pure answer-frequency signal, so a consistently wrong consensus can receive the same high soft reward as a correct consensus; in that regime the masked objective would stably reinforce reasoning that produces the wrong answer. The empirical consensus-accuracy curves in Figure 6 and the weak-initialization and low-contamination experiments mitigate this concern for the tested distributions, and the Limitations section concedes the reliance on outcome-based signals. However, the main text phrase "provably eliminated" (Section 3.2) and the wording of Theorem 2 overstate the theoretical reach. Please state the correctness assumption explicitly, soften the proof claims accordingly, or add a formal treatment of the wrong-consensus case.
  4. [Section 3.3 and Appendix E.7] CAR uses the policy being trained as its own pairwise judge: the augmented answer pool is generated by the same model whose weights are being updated. The mechanistic analysis in Appendix E.7 shows that position bias decreases and that correct trajectories produce lower pairwise entropy, which is useful evidence. However, it does not include a control against an independent judge or a fixed reference policy over the full training run, so it remains possible that the comparison branch itself drifts with the policy. Please add such a control or clearly state that the pairwise comparisons are assumed to remain faithful throughout training.
  5. [Section 4.2 and footnote 3] The term "avg@k setting" is used for the main result, but the metric is not defined in the main text; it is only inferred from Appendix A that each benchmark uses a different k (e.g., avg@16 for AIME, avg@4 for MATH500 and GSM8K, avg@8 for AMC). Please define the aggregate metric in Section 4.2 and clarify whether the reported averages in Table 1 are unweighted means across benchmarks with different k values.
minor comments (5)
  1. [Introduction] In the second paragraph of Section 1, "Majority V oting" contains a stray space; this also appears in the baseline list in Section 4.1.
  2. [Appendix D] The appendix title uses "Overcome-Masked Update" (in D.2) while the main text consistently uses "Outcome-Masked Update"; please unify the terminology.
  3. [Appendix E.2] The Table 5 caption reads "Pass@k Results (%) ofRL performancecomparison" and should be spaced and reworded for readability.
  4. [Figure 9] The caption and the in-text discussion in Appendix E.7 refer to panels (a)-(d), but the figure layout and the order of discussion make it hard to map the panels; please align the panel labels with the textual references.
  5. [Section 4.4] The ablation paragraph reports average drops for "w/o OMU" on only two of the three backbones; for completeness, please report the corresponding drop for Llama-3.2-3B-Instruct as well.

Circularity Check

1 steps flagged · score 1.0 of 10

No load-bearing circularity: the 'matches GT-reward' claim is benchmark-empirical; the only definitional move is Appendix D.2's proof that masked answer tokens have zero gradient, which is the mask definition restated.

  1. self definitional [Appendix D.2, Lemma 5/Theorem 2; mask defined in Eq. (1), Section 3.2]
    "Lemma 5. Under the masked objective (20), for any sample i and any position t such that mi,t = 0, ∂Lmask/∂logπθ(τi,t |x, τi,<t) = 0. Proof. By construction, neither the policy-gradient term nor the KL regularization term in (20) contains any factor involving logπθ(τi,t |x, τi,<t) for t > T."

    The result is true by construction: m_{i,t} was defined in Eq. (1) as 0 for answer-span tokens, so the loss in Eq. (20) contains no answer-token log-probability terms. Presenting this as a 'theoretical proof' that masking redirects credit restates the definition of the mask rather than deriving a consequence. Theorem 2 extends this with a KL-barrier argument for stability, but the further claim that optimization must 'reliably support correct answers' does not follow from the masked objective alone, since the majority-vote reward in Appendix D.2 does not encode correctness. This step is non-load-bearing for the paper's central empirical claim, which is tested against external benchmarks.

full rationale

OM-GRPO's headline result—label-free RLVR that 'matches supervised GT-reward training'—is an empirical claim. It is supported by Table 1 comparisons with GT-Reward and label-free baselines, component ablations (Table 3), 10-epoch stability curves (Figure 3), TTRL results (Table 2), Pass@k results, low-contamination benchmarks (Table 6), and a ReCEval-style reasoning-quality evaluation (Table 7). These are external benchmarks with no fitted constants re-labeled as predictions and no load-bearing self-citation; the cited MV, Entropy, Self-Certainty, and CoReward baselines are prior work by other groups. The only definitional moment is Appendix D.2, where Lemma 5 states that masked answer tokens have zero gradient—exactly what the mask m_{i,t}=0 in Eq. (1) means—so the 'theory' restates its own construction. Theorem 2 contributes a KL-barrier stability argument but does not prove that group consensus is correct; the paper's Limitations section explicitly concedes that OM-GRPO 'still relies on outcome-based reward signals rather than explicit supervision of reasoning quality.' Because the main claim is independently benchmark-tested and the tautological lemma is not the sole evidence, the circularity is minor and non-load-bearing.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The method introduces no new physical or mathematical entities; all components (soft rewards, masks, pairwise comparisons) are algorithmic choices. The main external grounding is the open benchmarks and backbones used for evaluation.

free parameters (2)
  • Group size G = 8
    Number of sampled trajectories per question used to estimate soft answer-frequency rewards. The reliability of the label-free signal depends on this choice; fixed by hand, not fitted.
  • KL coefficient beta = 0.005
    Regularization strength in the GRPO objective (Eq. 1). Chosen by hand; affects the stability-analysis claims in Appendix D.
assumptions (3)
  • domain assumption The extracted answer span inside \boxed{} is present and parseable for essentially all training rollouts, and the format reward correctly identifies parseable outputs.
    Used in Section 3.1 to define the format reward and the answer span z; the method's masking and reward estimation require reliable extraction.
  • domain assumption Answer-frequency consensus within a small sampled group is positively correlated with answer correctness for the training distribution.
    Central premise of label-free RLVR; introduced in Section 3.1 Eq. (2) and revisited in Section 4.6. If consensus is wrong, optimization targets a wrong answer even with masking.
  • ad hoc to paper The simplified categorical softmax model in Appendix D (Eqs. 5-6) captures the relevant collapse dynamics of the transformer policy.
    The proofs in D.1 and D.2 model answer logits as linear in a shared hidden state with no input-dependent structure; this is a much simpler model than the actual transformer, and the paper does not justify that the simplified dynamics transfer.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Don't Peek at the Answer: Outcome-Masked Group Relative Policy Optimization for Label-Free RLVR." pith.science (2026). https://pith.science/paper/UU7TUH2F

@misc{pith2026260803119,
  author       = {Pith},
  title        = {Pith review of: Don't Peek at the Answer: Outcome-Masked Group Relative Policy Optimization for Label-Free RLVR},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UU7TUH2F}},
  note         = {Machine review of arXiv:2608.03119}
}
read the original abstract

Reinforcement Learning with Verifiable Rewards (RLVR) improves LLM reasoning but typically relies on ground-truth (GT) answers, limiting scalability. Voting-based label-free RLVR replace gold supervision with answer-level consensus from model samples. However, collapse arises when the same answer-level signal is used both to estimate rewards and to drive token-level policy optimization, encouraging the model to directly reinforce answer tokens rather than improve reasoning. We propose OM-GRPO, a label-free RLVR framework that decouples reward estimation from policy optimization. OM-GRPO masks gradients on the answer span while retaining answer-level rewards through a soft consensus signal, shifting optimization pressure away from answer tokens. We further introduce Contrast-Augmented Reward, which refines reward estimation via low-cost pairwise comparisons over existing trajectories without additional rollouts. Across diverse reasoning benchmarks and three LLM backbones, OM-GRPO consistently outperforms existing label-free RLVR methods and matches supervised GT-reward training with stable optimization. This stability is particularly beneficial in the Test-Time Training setting, where OM-GRPO surpasses majority voting by 4.24 points.

Figures

Figures reproduced from arXiv: 2608.03119 by the authors.

Figure 1
Figure 1. Training dynamics of different method on MATH5000 validation set. Left and middle: validation accuracy [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of OM-GRPO with outcome masking updates and contrast-augmented reward estimation. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Training dynamics on the MATH5000 validation set, with the KL loss scaled by the KL coefficient. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: MATH5000 validation accuracy under softer [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: KL divergence of answer tokens and reasoning [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Impact of CAR on MATH5000 validation accuracy over training steps. [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Prompt template for CAR. • MATH500 and GSM8K (Lightman et al., 2024; Cobbe et al., 2021). We report avg@4 accuracy for both benchmarks, computed by averaging correctness over 4 independently sampled solutions. Evaluation is performed with lighteval. 5 • AMC (math-ai Te…
Figure 8
Figure 8. Figure 8: Average unique answers per sample over train [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: Mechanistic analysis of CAR. CAR improves reward estimation by amplifying reliable pairwise signals [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]
Figure 10
Figure 10. Figure 10: Rollout-level analysis on Qwen3-1.7B-Base, including top-answer share, accuracy, mean training reward, [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]
Figure 11
Figure 11. Figure 11: Math5000 validation accuracy of PRM-based [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]
Figure 12
Figure 12. Figure 12: Effect of base-model strength on label-free RLVR. [PITH_FULL_IMAGE:figures/full_fig_p022_12.png]
Figure 14
Figure 14. Figure 14: Test-Time Training dynamics on the open [PITH_FULL_IMAGE:figures/full_fig_p022_14.png]
Figure 15
Figure 15. Figure 15: A Case Illustrating Reasoning–Answer Mismatch. [PITH_FULL_IMAGE:figures/full_fig_p023_15.png]
Figure 16
Figure 16. Figure 16: A Case Demonstrating Reasoning–Answer Consistency Achieved by OM-GRPO. [PITH_FULL_IMAGE:figures/full_fig_p024_16.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

40 extracted references · 7 canonical work pages

  1. [1]

    Shivam Agarwal, Zimin Zhang, Lifan Yuan, Jiawei Han, and Hao Peng. 2025. https://doi.org/10.48550/ARXIV.2505.15134 The unreasonable effectiveness of entropy minimization in llm reasoning . CoRR, abs/2505.15134

    arXiv doi:10.48550/arxiv.2505.15134 2025

  2. [2]

    Jie Cheng, Gang Xiong, Ruixi Qiao, Lijun Li, Chao Guo, Junle Wang, Yisheng Lv, and Fei-Yue Wang. 2025. https://arxiv.org/abs/2504.15275 Stop summation: Min-form credit assignment is all process reward model needs for reasoning . Preprint, arXiv:2504.15275

    arXiv 2025

  3. [3]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. https://doi.org/10.48550/ARXIV.2110.14168 Training verifiers to solve math word problems . CoRR, abs/2110.14168

    arXiv doi:10.48550/arxiv.2110.14168 2021

  4. [4]

    DeepSeek - AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, and 81 others. 2025. https://doi.org/10.48550/ARXIV.2501.12948 Deepseek-r1: Incentivizing reasoning capability in llms via reinfor...

    arXiv doi:10.48550/arxiv.2501.12948 2025

  5. [5]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al - Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, and et al. 2024. https://doi.org/10.48550/ARXIV.2407.21783 The llama 3 her...

    arXiv doi:10.48550/arxiv.2407.21783 2024

  6. [6]

    Alex Gu, Baptiste Rozi \`e re, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I. Wang. 2024. https://doi.org/10.48550/ARXIV.2401.03065 Cruxeval: A benchmark for code reasoning, understanding and execution . CoRR, abs/2401.03065

    arXiv doi:10.48550/arxiv.2401.03065 2024

  7. [7]

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/be83ab3ecd0db773eb2dc1b0a17836a1-Abstract-round2.html Measuring mathematical problem solving with the MATH dataset . In Proceedings of the Neural Information Processin...

    work page 2021

  8. [8]

    Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung - Yeung Shum. 2025. https://doi.org/10.48550/ARXIV.2503.24290 Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model . CoRR, abs/2503.24290

    arXiv doi:10.48550/arxiv.2503.24290 2025

Show all 40 references
  1. [9]

    Hugging Face H4 . 2024. AIME 2024 Benchmark . https://huggingface.co/datasets/HuggingFaceH4/aime_2024

    2024

  2. [10]

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennett, Ananya K...

    2024 arXiv

  3. [11]

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024. https://doi.org/10.48550/ARXIV.2403.07974 Livecodebench: Holistic and contamination free evaluation of large language models for code . Co...

    2024 doi arXiv

  4. [12]

    Kimi-Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, Chuning Tang, Congcong Wang, Dehao Zhang, Enming Yuan, Enzhe Lu, Fengxiang Tang, Flood Sung, Guangda Wei, Guokun Lai, and 75 others. 2025. https://doi...

    2025 doi arXiv

  5. [13]

    Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luc...

    2024 doi arXiv

  6. [14]

    Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, and Sushant Prakash. 2024. https://arxiv.org/abs/2309.00267 Rlaif vs. rlhf: Scaling reinforcement learning from human feedback wit...

    2024 arXiv

  7. [15]

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. https://doi.org/10.48550/ARXIV.2305.20050 Let\'s verify step by step . CoRR, abs/2305.20050

    2024 doi arXiv

  8. [16]

    math-ai Team . 2024. Amc23: American mathematics competitions 2023 test set. https://huggingface.co/datasets/math-ai/amc23. Hugging Face dataset; 40 AMC 2023 problems; accessed 2025-09-16

    2024

  9. [17]

    OpenCompass . 2025. AIME 2025 Benchmark . https://huggingface.co/datasets/opencompass/AIME2025

    2025

  10. [18]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and...

    2022 doi arXiv

  11. [19]

    Jing - Cheng Pang, Pengyuan Wang, Kaiyuan Li, Xiong - Hui Chen, Jiacheng Xu, Zongzhang Zhang, and Yang Yu. 2023. https://doi.org/10.48550/ARXIV.2305.14483 Language model self - improvement by reinforcement learning contemplation . CoRR, abs/2305.14483

    2023 doi arXiv

  12. [20]

    Mihir Prabhudesai, Lili Chen, Alex Ippoliti, Katerina Fragkiadaki, Hao Liu, and Deepak Pathak. 2025. https://doi.org/10.48550/ARXIV.2505.22660 Maximizing confidence alone improves reasoning . CoRR, abs/2505.22660

    2025 doi arXiv

  13. [21]

    Archiki Prasad, Swarnadeep Saha, Xiang Zhou, and Mohit Bansal. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.622 R e CE val: Evaluating reasoning chains via correctness and informativeness . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Pro...

    2023 doi

  14. [22]

    Manning, Stefano Ermon, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. 2023. http://papers.nips.cc/paper\_files/paper/2023/hash/a85b405ed65c6477a4fe8302b5e06ce7-Abstract-Conference.html Direct preference optimization: Your language model is sec...

    2023

  15. [23]

    Sheikh Shafayat, Fahim Tajwar, Ruslan Salakhutdinov, Jeff Schneider, and Andrea Zanette. 2025. https://doi.org/10.48550/ARXIV.2505.21444 Can large reasoning models self - train? CoRR, abs/2505.21444

    2025 doi

  16. [24]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. https://doi.org/10.48550/ARXIV.2402.03300 Deepseekmath: Pushing the limits of mathematical reasoning in open language models . CoRR, abs/2...

    2024 doi arXiv

  17. [25]

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. 2025. https://doi.org/10.1145/3689031.3696075 Hybridflow: A flexible and efficient RLHF framework . In Proceedings of the Twentieth European Conference on Compute...

    2025

  18. [26]

    Yi Su, Dian Yu, Linfeng Song, Juntao Li, Haitao Mi, Zhaopeng Tu, Min Zhang, and Dong Yu. 2025. https://doi.org/10.48550/ARXIV.2503.23829 Crossing the reward bridge: Expanding RL with verifiable rewards across diverse domains . CoRR, abs/2503.23829

    2025 doi arXiv

  19. [27]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton - Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu,...

    2023 doi arXiv

  20. [28]

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. https://openreview.net/forum?id=pZ3i2yt5DoY Self-consistency improves chain of thought reasoning in language models . In International Conference on Learning...

    2023

  21. [29]

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. 2024. https://doi.org/10.48550/ARXIV.2406.01574 Mmlu-pro: A more r...

    2024 doi arXiv

  22. [30]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2022. https://doi.org/10.48550/ARXIV.2201.11903 Chain - of - thought prompting elicits reasoning in large language models . CoRR, abs/2201.11903

    2022 doi arXiv

  23. [31]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, and et al. 2025 a . https://doi.org/10.48550/ARXIV.2505.09388 Qwen3 te...

    2025 doi arXiv

  24. [32]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, and 24 others. 2025 b . https://doi.org/10....

    2025 doi arXiv

  25. [33]

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Hongchang Gao, Arman Kulkarni, Binyuan Hui, Lei Li, Zhengyuan Xu, Minhao Jiang, Qi Liu, Xiang Zhu, Di Chen, Longjun Li, Wensen Cheng, Yu Zhang, Tianqi Zhang, Hao Zhang, Chao Shang, and 15 others. 2025 a . https://doi.org/10.4855...

    2025 doi arXiv

  26. [34]

    Zhaoning Yu, Will Su, Leitian Tao, Haozhu Wang, Aashu Singh, Hanchao Yu, Jianyu Wang, Hongyang Gao, Weizhe Yuan, Jason Weston, Ping Yu, and Jing Xu. 2025 b . https://arxiv.org/abs/2510.02172 Restrain: From spurious votes to signals -- self-driven rl with self-penalization . Pr...

    2025

  27. [35]

    Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. 2025. https://arxiv.org/abs/2504.13837 Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? Preprint, arXiv:2504.13837

    2025 arXiv

  28. [36]

    Zizhuo Zhang, Jianing Zhu, Xinmu Ge, Zihua Zhao, Zhanke Zhou, Xuan Li, Xiao Feng, Jiangchao Yao, and Bo Han. 2025. https://doi.org/10.48550/ARXIV.2508.00410 Co - reward: Self - supervised reinforcement learning for large language model reasoning via contrastive agreement . CoR...

    2025 doi

  29. [37]

    Andrew Zhao, Yiran Wu, Yang Yue, Tong Wu, Quentin Xu, Matthieu Lin, Shenzhi Wang, Qingyun Wu, Zilong Zheng, and Gao Huang. 2025 a . https://doi.org/10.48550/ARXIV.2505.03335 Absolute zero: Reinforced self - play reasoning with zero data . CoRR, abs/2505.03335

    2025 doi arXiv

  30. [38]

    Xuandong Zhao, Zhewei Kang, Aosong Feng, Sergey Levine, and Dawn Song. 2025 b . https://doi.org/10.48550/ARXIV.2505.19590 Learning to reason without external rewards . CoRR, abs/2505.19590

    2025 doi arXiv

  31. [39]

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. https://doi.org/10.48550/ARXIV.2311.07911 Instruction-following evaluation for large language models . CoRR, abs/2311.07911

    2023 doi arXiv

  32. [40]

    Jiaru Zou, Ling Yang, Jingwen Gu, Jiahao Qiu, Ke Shen, Jingrui He, and Mengdi Wang. 2025. https://arxiv.org/abs/2506.18896 Reasonflux-prm: Trajectory-aware prms for long chain-of-thought reasoning in llms . Preprint, arXiv:2506.18896

    2025

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.