Pith. sign in

REVIEW 3 major objections 6 minor 60 references

This paper claims that injecting weak-model 8-token branches into a stronger model's reasoning prefixes outperforms matched-budget GRPO on math benchmarks while cutting training time 3.55×.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 14:40 UTC pith:N5T4YVRZ

load-bearing objection A plausible, cleanly specified method for cheap local exploration in reasoning RL, but the headline accuracy gain over GRPO sits inside the noise band and needs better statistical evidence before it is taken as established. the 3 major comments →

arxiv 2607.16205 v1 pith:N5T4YVRZ submitted 2026-05-07 cs.AI

It Takes 8 Tokens: Weak-to-Strong Off-Policy RL via Auxiliary Branches

Dayu Wang , Jiaye Yang , Weikang Li , Jiahui Liang , Liwei Qian , Xin Pei , Jizhou Huang This is my paper
classification cs.AI
keywords reinforcement learningverifiable rewardsweak-to-strongoff-policy RLauxiliary branchesreasoning diversityGRPOcredit assignment
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that the main obstacle to outcome-reward RL for reasoning is not sampling budget but support: a model's rollouts cluster into the same wrong reasoning basins, giving little reward contrast. W2SPO attacks this by having a frozen, weaker auxiliary model propose eight-token branch continuations at an intermediate point in the target model's own trajectory; the target model then finishes the solution, and the verifier reward updates only those eight branch tokens. On math benchmarks at the 4-billion-parameter scale, the method raises greedy Pass@1 from 62.3% to 64.2% relative to matched-sampling GRPO and reports a 3.55× total training speedup, while better preserving Pass@k sampling performance. If the mechanism is real, it means a weaker model can usefully steer a stronger one's exploration without being able to solve the problem itself.

Core claim

The central claim is that short auxiliary branches—often just 8 tokens—sampled from a weaker, frozen model and inserted into an intermediate prefix of the stronger target model expand the local support of the rollout distribution enough to improve RL with verifiable rewards. Concretely, W2SPO builds A-B-A trajectories: the target generates a prefix, the auxiliary proposes a short branch, and the target completes the reasoning path. Policy gradients are confined to the inserted branch tokens, with a tempered importance weight and leave-one-out advantage computed from the final verifier reward. The paper reports that this branch-level, off-policy update outperforms a matched-sampling GRPO base

What carries the argument

The load-bearing object is the auxiliary branch: an 8-token continuation sampled from a frozen weaker model (with actor-sampled branches as controls), inserted after a 50-token greedy prefix. Its value is estimated by letting the target model roll out K=2 completions from the branched context and averaging the verifier reward. A tempered, clipped importance coefficient balances off-policy auxiliary proposals against actor proposals, a weighted leave-one-out baseline computes branch advantages, and the PPO-clipped objective is masked so gradients touch only the branch tokens. This turns the update into a local policy-gradient step on a stop-gradient downstream value.

Load-bearing premise

The assumption that a branch's value is faithfully captured by averaging just two completions—so branch rewards take only the values 0, 0.5, and 1—and that the resulting noisy advantages still carry a real training signal rather than just noise.

What would settle it

Rescore the same branch groups with 16 or 32 completions per branch and re-run W2SPO with several seeds; if the Pass@1 advantage over matched-sampling GRPO collapses or flips sign, the two-rollout branch score was the load-bearing assumption. A second check: replace auxiliary branches with random 8-token text from an arbitrary source; if performance matches W2SPO, the gain comes from token insertion alone and not from weak-model proposals.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Reasoning RL does not have to choose between self-generated rollouts and external teachers: a weak model used only for local proposals can expand exploration while the target keeps long-horizon control.
  • If the Pass@1 gain is real, other self-rollout RL methods could adopt branch-level credit assignment to reduce the compute wasted on all-wrong groups.
  • The 3.55× speedup implies that at a fixed compute budget, branch-based RL can train larger models or more epochs than matched-sampling GRPO.
  • Better preservation of Pass@k after training suggests the method counteracts diversity collapse, mitigating the known shrinkage of the exploration frontier under RL.
  • Because updates touch only 8 tokens, the approach naturally localizes credit assignment to the decision point that changes downstream reasoning.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the auxiliary model may not need to be 'weaker'—it needs to be diverse; a stronger but differently distributed proposer might yield even larger support expansion, a hypothesis the paper does not test.
  • Editorial inference: the same branch mechanism could transfer to other verifiable-reward domains such as code generation or theorem proving, where semantic redundancy of rollouts is also a bottleneck.
  • Editorial inference: the 2-rollout branch score is a very coarse estimator; pairing branch-value estimation with learned value models or more rollouts could sharpen the advantage signal and presumably improve stability.
  • Editorial inference: the speedup comes partly from not backpropagating through long continuations; this suggests an efficiency-accuracy tradeoff curve that the paper only samples at one point (Lb=8).

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes W2SPO, an off-policy RL method for reasoning LLMs. The actor first generates a fixed greedy prefix, then short branches (typically 8 tokens) are sampled from both the actor and a frozen weaker auxiliary model; the actor completes the trajectory from each branch, a verifier scores the completed solutions, and the policy is updated only on the branch tokens using an importance-weighted, clipped group advantage. The paper argues that this expands local rollout support and claims that, under a matched sampling budget, W2SPO improves greedy Pass@1 over GRPO from 62.3% to 64.2% on a 1,763-problem math suite while achieving a 3.55x training speedup, and that it better preserves multi-sample Pass@k performance.

Significance. If the empirical claims hold, W2SPO is a meaningful contribution: it identifies a plausible support-limitation failure mode in self-rollout RL, proposes a concrete and clearly specified remedy, and reports a large training-speedup benefit that is mechanically credible because gradients are applied only to the short branch window. The paper's strengths include well-documented method details, motivated diversity diagnostics, controlled ablations, robustness checks with a second seed and a second auxiliary model, and an anonymized code release. The theoretical analysis is also honestly scoped as a local semi-gradient approximation rather than a convergence guarantee. However, the headline accuracy advantage over GRPO is currently not statistically supported: the reported difference is within the noise band of the evaluation set, and no confidence intervals or significance tests are provided for any accuracy comparison. The contribution is therefore promising but its central empirical claim needs substantially stronger evidence.

major comments (3)
  1. [§5.2, Table 1(a)] The central claim that W2SPO outperforms matched-sampling GRPO rests on 64.2% vs. 62.3% on 1,763 problems. A binomial standard error for this difference is approximately 1.6 percentage points, so the observed 1.9-point gap is only about 1.2 standard errors and is not significant at conventional levels. The per-benchmark gaps on AIME24 and AIME25 are based on roughly 30 problems each and correspond to one or two problems. The robustness run with seed=2026 (63.9%, Table 3) lies inside the same noise band. No confidence intervals, bootstrap estimates, or paired tests are reported for any accuracy number, and the matched-sampling GRPO baseline is run with a single seed. This does not yet establish the headline superiority claim; please provide multi-seed results for both methods, paired or stratified significance tests, or bootstrap CIs.
  2. [§4, Step 3] The branch reward is defined as r_i = (1/K) Σ_k R(·) with K=2, so r_i takes values only in {0, 0.5, 1}. Combined with the small group size, the resulting branch-level advantages are extremely coarse, and no training-time diagnostics (e.g., the ESS values proposed in Appendix B) are reported. This is a plausible source of additional variance that could account for part of the observed accuracy gap independent of the method's intended mechanism. Please quantify the variance of the branch-value estimator, report the proposed diagnostics, or otherwise demonstrate that the advantage signal is not dominated by this noise.
  3. [§5.5, Table 2] The ablation claims that both cross-model branch diversity and importance weighting are 'critical' rely on differences of 1.6–2.3 points (e.g., A-only 61.9% vs. W2SPO 64.2%) without any error bars. Under the same binomial noise as the main comparison, these differences are not clearly significant. If the paper is to claim that the ablations validate the mechanism, it should report confidence intervals or multiple seeds for the ablated variants as well.
minor comments (6)
  1. [NeurIPS Checklist, item 7] The checklist answers [Yes] to 'experiment statistical significance' and justifies it with robustness checks, but no error bars or significance tests are given for benchmark accuracy. This is inconsistent; either add such statistics or answer [No] and explain the limitation.
  2. [§5.2 and Table 1(a)] Please state the exact number of problems in each benchmark subset (especially AIME24/AIME25) in the main text. Percentages on ~30-problem sets are coarse; reporting counts would prevent overinterpretation of per-benchmark differences.
  3. [§3, Figure 3] The 'hard subset' used in the insertion-position experiment is described only in passing, with no sample size or selection criterion. Please define this subset precisely.
  4. [Appendix B, Eq. (20)] The clipping bound is written as [−M, M] but M is not defined in the appendix; the main text uses 30. Please reconcile the notation.
  5. [Checklist, item 7 justification] Minor typo: 'standard divisions' should be 'standard deviations.'
  6. [§6 / Appendix B] The local policy-gradient derivation is clearly labeled as a stop-gradient approximation. The paper should also state explicitly in the main text that the importance weights are not unbiased, which it does in Appendix B; consider moving a one-sentence version of this caveat into Section 4 for readers who do not read the appendix.

Circularity Check

0 steps flagged

No significant circularity: W2SPO's method derivation is a standard semi-gradient identity, its configuration is selected on motivation data rather than the evaluation targets, and its main claims are external benchmark comparisons.

full rationale

The paper's derivation chain is not circular. Step 3 defines a branch reward r_i as the mean of K=2 verifier-scored rollouts; this is a noisy Monte Carlo estimate, but the policy update optimizes the branch-token log-probabilities against that reward and the headline Pass@1 numbers are measured on a held-out set of 1,763 problems, not reconstructed from the training rewards. Step 4 uses a tempered importance coefficient c_i; the appendix explicitly states it is not an unbiased importance-sampling ratio but a bounded bias-variance heuristic, so nothing is being relabeled as a first-principles result. The Section 6 local-policy-gradient equation (3) is the standard REINFORCE identity under an explicit stop-gradient assumption; it does not assume the conclusion that auxiliary branches help. The choice of Lb=8 and insertion position 50 comes from Figure 3's Pass@8 experiments on a hard subset, which is hyperparameter selection, not a fitted parameter being called a prediction; the final comparison to matched-sampling GRPO and ablated variants uses independent evaluation benchmarks. The only self-citation that overlaps with the authors (Wang et al. 2026) is used as background in Related Work and is not load-bearing; the weak-to-strong premise is independently supported by the paper's own A-B-A inference experiments and by external citations. The absence of confidence intervals for accuracy (NeurIPS checklist item 7) is a statistical-support concern, not a circularity concern. No equation in the paper reduces to its own inputs by construction.

Axiom & Free-Parameter Ledger

7 free parameters · 5 axioms · 0 invented entities

The method has no newly postulated physical or scientific entities. The load-bearing assumptions are about verifier reliability, coherent continuation after branch insertion, useful weak-model proposals, and the validity of the local semi-gradient approximation. The free parameters are mostly hand-chosen hyperparameters and training-data filters rather than constants fitted to a scientific target.

free parameters (7)
  • branch_length = 8 tokens
    Chosen from a sweep over B=8/16/32 in Figure 3; central to the method and title, but no principled derivation.
  • insertion_position = 50 tokens
    Chosen from a sweep over positions 50/100/200 in Figure 3; affects off-policy context and reported gains.
  • importance_temperature_alpha = 0.02
    Hand-selected scalar controlling the influence of off-policy auxiliary branches in the advantage estimator.
  • importance_cap_cmax = 2.0
    Hand-selected upper bound on branch importance weights; no sensitivity analysis reported.
  • branch_and_completion_counts = nA=2, nB=6, K=2
    Fixed sampling configuration; no sensitivity analysis for group composition or number of completions per branch.
  • semantic_diversity_threshold = tau=0.95
    Threshold used to count 'semantically distinct correct trajectories'; sensitivity only at 0.94 and 0.96.
  • training_data_filters = difficulty 12-15; Pass@8 in (0,8); 800 shortest zero-success labels
    Ad hoc construction of the training distribution; may inflate benchmark gains relative to training on the full data.
axioms (5)
  • domain assumption The rule-based verifier reward correctly identifies solution correctness.
    All credit assignment and evaluation depend on the final verifier; verifier noise from regex extraction or math-verify is not analyzed.
  • domain assumption After an 8-token auxiliary branch is inserted, the actor continues coherently and its completions provide a meaningful basis for branch value.
    Core A-B-A premise: if foreign branches derail generation, the branch reward estimate loses meaning.
  • domain assumption A weaker model's 8-token branches expand the actor's support and contain branches with higher downstream success probability.
    Necessary for auxiliary contribution; supported empirically on a few benchmarks but not guaranteed across tasks or scales.
  • ad hoc to paper The local policy-gradient identity in Section 6 holds because the continuation policy and Qbar are treated as independent of theta and the branch support is fixed.
    Appendix B explicitly states this is a semi-gradient approximation, not an unbiased estimator or a convergence guarantee.
  • ad hoc to paper The clipped, tempered importance coefficient provides a practical bias-variance tradeoff.
    Appendix B admits ci is not an unbiased importance ratio; its effectiveness is empirical rather than derived.

pith-pipeline@v1.3.0-alltime-deepseek · 21724 in / 11538 out tokens · 123156 ms · 2026-08-02T14:40:46.479923+00:00 · methodology

0 comments
read the original abstract

Reinforcement learning with verifiable rewards has emerged as a standard approach for enhancing reasoning in large language models, which typically optimizes the policy by contrasting multiple self generated rollouts. However, we identify a critical support limited bottleneck in this paradigm: on challenging reasoning tasks, the target model's samples often exhibit semantic redundancy, converging into the same erroneous "reasoning basins" that offer negligible reward contrast for policy updates. In this paper, we propose to overcome this limitation through a weak to strong learning paradigm, where a policy's exploration is informed by a weaker but computationally efficient auxiliary model. We introduce W2SPO, an off policy RL method that injects short auxiliary segments often as brief as 8 tokens into intermediate target model trajectories and the target model then completes the reasoning path from these diverted states. Policy updates are restricted to these short inserted segments based on final verifiable rewards. Empirically, W2SPO achieves superior performance among evaluated 4B scale models on mathematical reasoning benchmarks, outperforming evaluated post trained baselines. Compared with vanilla GRPO under the same sampling budget, W2SPO improves Pass@1 from 62.3% to 64.2% while achieving a 3.55 times training speedup. These results suggest that weak auxiliary branches can induce stronger target reasoning policies by expanding local exploration support.

Figures

Figures reproduced from arXiv: 2607.16205 by Dayu Wang, Jiahui Liang, Jiaye Yang, Jizhou Huang, Liwei Qian, Weikang Li, Xin Pei.

Figure 1
Figure 1. Figure 1: Overview of W2SPO. Standard GRPO relies on self-rollouts from the target model, which [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Self-rollout is diminishing-return. As the sampling budget [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Auxiliary branches expand effective reasoning support. Left: average number of seman [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 1
Figure 1. Figure 1: The method consists of five steps: greedy prefix generation, short branch proposal, rollout [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 5
Figure 5. Figure 5: Training-time reward-signal diagnostics for the controlled A-only branch ablation. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 4
Figure 4. Figure 4: Drop in Pass@k relative to the base model on AIME25. Smaller drops indicate better preservation of multi￾sampling performance after RL training. Beyond greedy Pass@1, we further evaluate whether RL training preserves the model’s exploration boundary un￾der large sampling budgets. Prior work has reported that reasoning RL can improve single-sample accuracy while degrading Pass@k, suggesting a shrinkage of t… view at source ↗
Figure 6
Figure 6. Figure 6: Semantic diversity of correct trajectories under similarity threshold [PITH_FULL_IMAGE:figures/full_fig_p014_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Semantic diversity of correct trajectories under similarity threshold [PITH_FULL_IMAGE:figures/full_fig_p015_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Drop in Pass@k relative to the base model on AIME24 This appendix gives a more explicit derivation of the local analysis in Section 6. The purpose is to justify the inter￾pretation of our update as a branch-level credit-assignment mechanism. We do not claim that the resulting algorithm is an unbiased estimator of the full off-policy policy gra￾dient, nor do we establish a convergence guarantee for the comp… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

60 extracted references · 22 linked inside Pith

  1. [1]

    2024 , eprint=

    DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models , author=. 2024 , eprint=

    2024

  2. [2]

    Guo, Daya and Yang, Dejian and Zhang, Haowei and Song, Junxiao and Wang, Peiyi and Zhu, Qihao and Xu, Runxin and Zhang, Ruoyu and Ma, Shirong and Bi, Xiao and Zhang, Xiaokang and Yu, Xingkai and Wu, Yu and Wu, Z. F. and Gou, Zhibin and Shao, Zhihong and Li, Zhuoshu and Gao, Ziyi and Liu, Aixin and Xue, Bing and Wang, Bingxuan and Wu, Bochao and Feng, Bei ...

  3. [3]

    2025 , eprint=

    Outcome-based Exploration for LLM Reasoning , author=. 2025 , eprint=

    2025

  4. [4]

    2025 , eprint=

    Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs , author=. 2025 , eprint=

    2025

  5. [5]

    2021 , eprint=

    Training Verifiers to Solve Math Word Problems , author=. 2021 , eprint=

    2021

  6. [6]

    2023 , eprint=

    Self-Consistency Improves Chain of Thought Reasoning in Language Models , author=. 2023 , eprint=

    2023

  7. [7]

    Scaling Self-Improving Foundation Models without Human Supervision , year=

    Assessing diversity collapse in reasoning , author=. Scaling Self-Improving Foundation Models without Human Supervision , year=

  8. [8]

    arXiv preprint arXiv:2504.13837 , year=

    Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? , author=. arXiv preprint arXiv:2504.13837 , year=

    Pith/arXiv arXiv

  9. [9]

    arXiv preprint arXiv:2407.21787 , year=

    Large language monkeys: Scaling inference compute with repeated sampling , author=. arXiv preprint arXiv:2407.21787 , year=

    Pith/arXiv arXiv

  10. [10]

    2026 , eprint=

    RL-PLUS: Countering Capability Boundary Collapse of LLMs in Reinforcement Learning with Hybrid-policy Optimization , author=. 2026 , eprint=

    2026

  11. [11]

    2026 , eprint=

    Stepwise Guided Policy Optimization: Coloring your Incorrect Reasoning in GRPO , author=. 2026 , eprint=

    2026

  12. [12]

    arXiv preprint arXiv:2602.05281 , year=

    Back to Basics: Revisiting Exploration in Reinforcement Learning for LLM Reasoning via Generative Probabilities , author=. arXiv preprint arXiv:2602.05281 , year=

    arXiv

  13. [13]

    arXiv preprint arXiv:2509.22230 , year=

    In Their Own Words: Reasoning Traces Tailored for Small Models Make Them Better Reasoners , author=. arXiv preprint arXiv:2509.22230 , year=

    arXiv

  14. [14]

    arXiv preprint arXiv:2510.25992 , year=

    Supervised reinforcement learning: From expert trajectories to step-wise reasoning , author=. arXiv preprint arXiv:2510.25992 , year=

    arXiv

  15. [15]

    arXiv preprint arXiv:2506.17211 , year=

    Bread: Branched rollouts from expert anchors bridge sft & rl for reasoning , author=. arXiv preprint arXiv:2506.17211 , year=

    Pith/arXiv arXiv

  16. [16]

    arXiv preprint arXiv:2510.09388 , year=

    Hint: Helping ineffective rollouts navigate towards effectiveness , author=. arXiv preprint arXiv:2510.09388 , year=

    Pith/arXiv arXiv

  17. [17]

    2023 , eprint=

    Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision , author=. 2023 , eprint=

    2023

  18. [18]

    arXiv preprint arXiv:2601.06160 , year=

    Student Guides Teacher: Weak-to-Strong Inference via Spectral Orthogonal Exploration , author=. arXiv preprint arXiv:2601.06160 , year=

    Pith/arXiv arXiv

  19. [19]

    Findings of the Association for Computational Linguistics: ACL 2023 , pages=

    Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes , author=. Findings of the Association for Computational Linguistics: ACL 2023 , pages=

    2023

  20. [20]

    Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024) , pages=

    Can small language models help large language models reason better?: LM-guided chain-of-thought , author=. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024) , pages=

    2024

  21. [21]

    Findings of the Association for Computational Linguistics: EMNLP 2024 , pages=

    Reasoning paths optimization: Learning to reason and explore from diverse paths , author=. Findings of the Association for Computational Linguistics: EMNLP 2024 , pages=

    2024

  22. [22]

    arXiv preprint arXiv:2504.12329 , year=

    Speculative thinking: Enhancing small-model reasoning with large model guidance at inference time , author=. arXiv preprint arXiv:2504.12329 , year=

    Pith/arXiv arXiv

  23. [23]

    Advances in neural information processing systems , volume=

    Tree of thoughts: Deliberate problem solving with large language models , author=. Advances in neural information processing systems , volume=

  24. [24]

    arXiv preprint arXiv:2506.13923 , year=

    Adaptive guidance accelerates reinforcement learning of reasoning models , author=. arXiv preprint arXiv:2506.13923 , year=

    Pith/arXiv arXiv

  25. [25]

    arXiv preprint arXiv:2503.14286 , year=

    Tapered off-policy reinforce: Stable and efficient reinforcement learning for llms , author=. arXiv preprint arXiv:2503.14286 , year=

    Pith/arXiv arXiv

  26. [26]

    arXiv preprint arXiv:2505.23564 , year=

    Segment policy optimization: Effective segment-level credit assignment in rl for large language models , author=. arXiv preprint arXiv:2505.23564 , year=

    arXiv

  27. [27]

    arXiv preprint arXiv:2504.14945 , year=

    Learning to reason under off-policy guidance , author=. arXiv preprint arXiv:2504.14945 , year=

    Pith/arXiv arXiv

  28. [28]

    arXiv preprint arXiv:2509.14257 , year=

    From correction to mastery: Reinforced distillation of large language model agents , author=. arXiv preprint arXiv:2509.14257 , year=

    Pith/arXiv arXiv

  29. [29]

    arXiv preprint arXiv:2509.16591 , year=

    From Uniform to Heterogeneous: Tailoring Policy Optimization to Every Token's Nature , author=. arXiv preprint arXiv:2509.16591 , year=

    Pith/arXiv arXiv

  30. [30]

    Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Treerl: Llm reinforcement learning with on-policy tree search , author=. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  31. [31]

    Advances in neural information processing systems , volume=

    Chain-of-thought prompting elicits reasoning in large language models , author=. Advances in neural information processing systems , volume=

  32. [32]

    arXiv preprint arXiv:2401.10774 , year=

    Medusa: Simple llm inference acceleration framework with multiple decoding heads , author=. arXiv preprint arXiv:2401.10774 , year=

    Pith/arXiv arXiv

  33. [33]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Math-shepherd: Verify and reinforce llms step-by-step without human annotations , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  34. [34]

    arXiv preprint arXiv:2410.08146 , year=

    Rewarding progress: Scaling automated process verifiers for llm reasoning , author=. arXiv preprint arXiv:2410.08146 , year=

    Pith/arXiv arXiv

  35. [35]

    arXiv preprint arXiv:2504.11343 , year=

    A minimalist approach to llm reasoning: from rejection sampling to reinforce , author=. arXiv preprint arXiv:2504.11343 , year=

    Pith/arXiv arXiv

  36. [36]

    arXiv preprint arXiv:2510.26122 , year=

    Reasoning Path Divergence: A New Metric and Curation Strategy to Unlock LLM Diverse Thinking , author=. arXiv preprint arXiv:2510.26122 , year=

    arXiv

  37. [37]

    arXiv preprint arXiv:2505.09655 , year=

    Dra-grpo: Exploring diversity-aware reward adjustment for r1-zero-like training of large language models , author=. arXiv preprint arXiv:2505.09655 , year=

    arXiv

  38. [38]

    Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Synergistic Weak-Strong Collaboration by Aligning Preferences , author=. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  39. [39]

    arXiv preprint arXiv:2501.17161 , year=

    Sft memorizes, rl generalizes: A comparative study of foundation model post-training , author=. arXiv preprint arXiv:2501.17161 , year=

    Pith/arXiv arXiv

  40. [40]

    arXiv preprint arXiv:2203.11171 , year=

    Self-consistency improves chain of thought reasoning in language models , author=. arXiv preprint arXiv:2203.11171 , year=

    Pith/arXiv arXiv

  41. [41]

    arXiv preprint arXiv:2510.08696 , year=

    Don't Waste Mistakes: Leveraging Negative RL-Groups via Confidence Reweighting , author=. arXiv preprint arXiv:2510.08696 , year=

    arXiv

  42. [42]

    arXiv preprint arXiv:2509.06941 , year=

    Outcome-based exploration for llm reasoning , author=. arXiv preprint arXiv:2509.06941 , year=

    Pith/arXiv arXiv

  43. [43]

    American Invitational Mathematics Examination (AIME) 2024 , author=

    2024

  44. [44]

    American Invitational Mathematics Examination (AIME) 2025 , author=

    2025

  45. [45]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  46. [46]

    arXiv preprint arXiv:2103.03874 , year=

    Measuring mathematical problem solving with the math dataset , author=. arXiv preprint arXiv:2103.03874 , year=

    Pith/arXiv arXiv

  47. [47]

    2024 , eprint=

    Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Models , author=. 2024 , eprint=

    2024

  48. [48]

    arXiv preprint arXiv:2505.22312 , year=

    Skywork Open Reasoner 1 Technical Report , author=. arXiv preprint arXiv:2505.22312 , year=

    Pith/arXiv arXiv

  49. [49]

    Skywork Open Reasoner Series , author =

  50. [50]

    2023 , eprint=

    Mistral 7B , author=. 2023 , eprint=

    2023

  51. [51]

    arXiv preprint arXiv:2604.01591 , year=

    ThinkTwice: Jointly Optimizing Large Language Models for Reasoning and Self-Refinement , author=. arXiv preprint arXiv:2604.01591 , year=

    Pith/arXiv arXiv

  52. [52]

    arXiv preprint arXiv:2503.14476 , year=

    Dapo: An open-source llm reinforcement learning system at scale , author=. arXiv preprint arXiv:2503.14476 , year=

    Pith/arXiv arXiv

  53. [53]

    arXiv preprint arXiv:2511.01937 , year=

    Shorter but not Worse: Frugal Reasoning via Easy Samples as Length Regularizers in Math RLVR , author=. arXiv preprint arXiv:2511.01937 , year=

    arXiv

  54. [54]

    arXiv preprint arXiv:2512.07461 , year=

    Native Parallel Reasoner: Reasoning in Parallelism via Self-Distilled Reinforcement Learning , author=. arXiv preprint arXiv:2512.07461 , year=

    Pith/arXiv arXiv

  55. [55]

    POLARIS: A Post-Training Recipe for Scaling Reinforcement Learning on Advanced Reasoning Models , url =

    An, Chenxin and Xie, Zhihui and Li, Xiaonan and Li, Lei and Zhang, Jun and Gong, Shansan and Zhong, Ming and Xu, Jingjing and Qiu, Xipeng and Wang, Mingxuan and Kong, Lingpeng , year =. POLARIS: A Post-Training Recipe for Scaling Reinforcement Learning on Advanced Reasoning Models , url =

  56. [56]

    arXiv preprint arXiv:2510.04996 , year=

    Reinforce-Ada: An Adaptive Sampling Framework under Non-linear RL Objectives , author=. arXiv preprint arXiv:2510.04996 , year=

    arXiv

  57. [57]

    2025 , eprint=

    Qwen3 Technical Report , author=. 2025 , eprint=

    2025

  58. [58]

    2023 , eprint=

    C-Pack: Packaged Resources To Advance General Chinese Embedding , author=. 2023 , eprint=

    2023

  59. [59]

    Gemma 3 , url=

    Gemma Team , year=. Gemma 3 , url=

  60. [60]

    2026 , eprint=

    Student Guides Teacher: Weak-to-Strong Inference via Spectral Orthogonal Exploration , author=. 2026 , eprint=

    2026