Pith. sign in

REVIEW 4 major objections 2 minor 85 references

Group reward variance controls GRPO's learning signal, so spending extra rollouts only on the highest-variance prompts reaches the same accuracy with up to 2.3x fewer rollouts, and raises final coding pass rates.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 06:06 UTC pith:DSQRVSKP

load-bearing objection A simple, credible variance-guided rollout allocator for GRPO; the theory is a bound and the m0=2 binary-variance issue is real but not fatal. the 4 major comments →

arxiv 2607.22002 v1 pith:DSQRVSKP submitted 2026-07-24 cs.AI

Learning as Reasoning Unfolds: Progressive Rollout Allocation for Efficient Reinforcement Learning

Heyang Jiang , Henry Liu , Baharan Mirzasoleiman This is my paper
classification cs.AI
keywords reinforcement learning with verifiable rewardsGRPOrollout allocationreward variancedata selectionLLM reasoningtraining efficiencychain-of-thought
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that when training reasoning LLMs with group-relative policy optimization and verifiable rewards, the most informative prompts are exactly those whose rollouts show high reward variance: the gradient magnitude for a prompt is bounded by a constant times the within-group reward standard deviation. It proposes VIGOR, which gives every prompt a tiny initial rollout budget and then, over several refinement rounds, concentrates additional rollouts on the prompts with the highest group reward variance until a fixed overall budget is used. Across math and coding benchmarks, this reaches the same or better accuracy with markedly fewer rollouts, up to 2.3x fewer on math and 1.49x fewer for coding's full pass rate, and adds a few points of final coding accuracy. The reason this matters is that rollout generation dominates RLVR training cost, so shifting computation toward the prompts that actually produce learning signal makes reasoning-model training cheaper and more stable, especially in the early epochs that drive most of the gain.

Core claim

The paper establishes that GRPO gradient magnitude is controlled by group reward variance. For binary rewards, it proves that the sum of absolute group-relative advantages equals G times the group reward standard deviation, and hence the norm of the KL-free GRPO gradient is at most M times sigma. It then turns this bound into an allocation rule: an iterative procedure that starts with m0=2 rollouts per prompt, keeps the top-alpha prompts by empirical reward variance, expands their rollout budget by a factor gamma, and repeats. Under a Pareto model for prompt-level variance, the resulting timescale speedup over uniform allocation grows with the number of refinement rounds and is larger for he

What carries the argument

The central identity is Theorem 1: with binary rewards and bounded importance-ratio gradients, the GRPO gradient norm satisfies the bound where sigma is the within-group reward standard deviation; the proof uses the equality for the sum of absolute z-scored advantages. This turns 'which prompt deserves more rollouts' into a measurable online statistic. The algorithm VIGOR around it performs iterative top-alpha selection by empirical group reward variance with multiplicative budget expansion gamma, leaving the GRPO update untouched. The supporting theoretical result (Theorem 2) predicts a closed-form speedup over GRPO that grows like (alpha^{-1/k})^{T/3} under Pareto-distributed prompt varian

Load-bearing premise

The load-bearing premise is that the reward variance estimated from an initial pool of just two rollouts per prompt reliably ranks which prompts would benefit from more rollouts, and that this ranking stays stable enough across refinement rounds and across tasks (math and coding) to concentrate the budget on prompts that actually drive learning.

What would settle it

Take a trained VIGOR run and, for prompts that VIGOR assigned to the low-budget track after its initial two rollouts, generate eight measurement rollouts without changing the allocation. If a substantial fraction, well above the reported 7.9%, of these low-budget prompts turn out to have high reward variance and high learning progress, the variance estimate is misordering prompts and the allocation advantage would be expected to degrade toward random selection. More directly: an oracle run that always allocated budget using eight-rollout variance estimates should beat VIGOR's two-rollout estim

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • VIGOR reaches GRPO's final coding full pass rate with 1.49x fewer rollouts and improves the average test pass rate by 3.4 points (63.4 to 66.8).
  • On math, VIGOR requires up to 2.3x fewer rollouts to reach target accuracy on Qwen2.5-3B, with consistent rollout-efficiency gains on 1.5B/7B and Phi-4-Mini, and the best rollout-matched average scores on all four model scales.
  • The wall-clock advantage is meaningful: on Qwen2.5-3B, VIGOR reaches the target accuracy in 50 steps versus 105 for GRESO, a 2.65x wall-clock speedup, with no extra generation or verification time at the same rollout budget.
  • VIGOR reduces the fraction of ineffective (zero-advantage) rollouts and creates an emergent curriculum: the rollout-weighted difficulty of selected prompts rises over training without explicit difficulty labels or a hand-designed schedule.
  • Because it leaves the GRPO update unchanged, variance-based allocation can be dropped into existing RLVR pipelines, matching the same total rollout budget per step as baselines (8 rollouts per prompt vs VIGOR's T=4, m0=2, gamma=2, alpha=0.5).

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If reward variance is an online proxy for a prompt's current learning frontier, variance-based allocation could be combined with other signals, such as policy entropy or success prediction, for prompts where two rollouts give a noisy variance estimate; the paper's reported 7.9% false-negative rate on the initial estimate suggests headroom for a hybrid rule.
  • The closed-form speedup under a Pareto model predicts that VIGOR's advantage grows as prompt variance becomes more heavy-tailed; a direct test would measure the empirical distribution of group reward variances across datasets and correlate tail heaviness with the measured speedup.
  • Because the theory is binary-reward, a natural extension is dense rewards, under which the gradient-variance link may be weaker or require a different normalization; testing whether variance-guided allocation still dominates in rich-feedback settings beyond full pass/fail would show how far the identity carries.
  • VIGOR's variance-based selection behaves like active learning or self-paced curriculum, suggesting it could transfer to other RL settings with verifiable outcomes, such as tool use or theorem proving, where rollout cost is high.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 2 minor

Summary. The paper proposes VIGOR, an iterative rollout allocation method for RLVR/GRPO. Instead of allocating a fixed rollout budget per prompt, VIGOR starts with m0=2 rollouts per prompt, computes reward variance, and iteratively prunes to the top-α fraction while expanding the rollout budget for retained prompts until a fixed total budget is reached. The authors prove (Theorem 1) that for binary rewards and KL-free GRPO, the gradient norm is bounded by Mσ, motivating variance as an informativeness signal. They further claim (Theorem 2) a closed-form speedup ratio under a Pareto model of prompt variance. Experiments on six math benchmarks (Qwen2.5-1.5B/3B/7B, Phi-4-Mini) and LiveCodeBench v6 (Qwen3-8B) report up to 2.3× rollout reduction on math and 1.49× on coding, plus gains in final accuracy. The empirical protocol matches rollout budgets across methods and includes ablations against difficulty-based and random selection.

Significance. If the empirical results are robust, VIGOR is a simple and practical contribution: it requires no auxiliary models, no extra rollout oversampling, and preserves the GRPO update. The comparisons against GRPO, GRESO, RL-ZVP, and PODS are extensive, with matched rollout budgets and multiple seeds. The strength of the paper is its experimental evaluation and the clarity of the method. However, the theoretical claim of a speedup guarantee is not supported by the current derivation (see major comments), and the reliability of the m0=2 variance signal is not yet established across tasks. The paper would be stronger if the theory is corrected or downgraded to a heuristic and the missing robustness analyses are added.

major comments (4)
  1. [Appendix D.2 / Theorem 2] The derivation of the speedup ratio is not valid. Eq. (23) turns the lower bound of Razin et al. (2025) into an equality τ = C·Var^{-1/3} with no justification; a lower bound on a hitting time does not determine its actual scaling. More fundamentally, Eq. (20) defines the 'variance contribution' of round i as γ^i m0 ∫_{u_i^*}^∞ u f(u) du, i.e., the per-prompt variance multiplied by the number of rollouts allocated. But in the GRPO objective each prompt contributes one term regardless of group size, so the optimization timescale depends on the prompt-level variance distribution, not on a rollout-weighted sum. Consequently η_exact and the ratios (4)-(5) do not follow from the stated assumptions. Please either supply a correct derivation or reframe the result as a heuristic.
  2. [§5.2 / Fig. 1] The headline '2.3× fewer rollouts' is not reproducible from the text: no target accuracy is given in the main text or figure; only later is a '37.8% target' mentioned for a specific wall-clock comparison. Speedup estimates are highly sensitive to the chosen target level. Please state the exact target accuracy for each reported speedup, specify how it was selected (ideally a priori), and include sensitivity analysis (e.g., speedup vs. target). Also clarify whether the rollout-efficiency curves use final or best checkpoints and how the speedup is interpolated.
  3. [§4.2/Algorithm 1, Appendix E.2] The method's core signal is the within-group variance estimated from m0=2 rollouts. For the binary-reward setting (math), this estimate can only distinguish 'mixed' from 'uniform'; it cannot rank prompts within the mixed group, so the first-round top-α selection is partly arbitrary. Appendix E.2 reports a single 7.9% false-negative rate for one Qwen2.5-3B MATH run; this does not measure rank correlation among mixed prompts, does not cover coding (partial-credit rewards), and does not examine stability over refinement rounds. Since pruned prompts receive no further rollouts in that training step, the paper should provide a more direct analysis of m0=2 reliability and should quantify the rollout-efficiency advantage of variance selection over the random-selection baseline (e.g., speedup ratios, not only accuracy curves).
  4. [§5.2 vs Appendix C] The number of training epochs is inconsistent: §5.2 says 'two epochs for the Qwen models and one epoch for Phi-4-Mini-Instruct,' while Appendix C says 'three epochs on MATH for the 1.5B and 3B models, and for two epochs on the MATH+DAPO mixture for the 7B model.' This affects the interpretation of Table 1 and must be reconciled.
minor comments (2)
  1. [§4.1 / Theorem 1] The theorem is an upper bound, but the text says variance 'directly governs' the gradient magnitude. Please soften the interpretation or provide a matching lower bound.
  2. [§2 and cross-references] There is a typo 'trainig' in the Related Work section. Also, Appendix D is described as containing proofs for 'Section 5', but Theorem 2 appears in Section 4.3; fix the cross-reference.

Circularity Check

0 steps flagged

No significant circularity: the theoretical bounds are derived from explicit assumptions and the efficiency gains are measured on independent benchmarks.

full rationale

The claimed derivation chain is self-contained rather than circular. Theorem 1 is obtained from Lemma 1 (triangle inequality on the clipped GRPO gradient) and Lemma 2 (exact binary-reward identity Σ|A_i|=Gσ); neither lemma presupposes the theorem's conclusion, and the bound ∥∇θL_GRPO∥≤Mσ is a genuine consequence, not a definition. VIGOR's use of reward variance as a selection score is motivated by this bound, but the bound is not an equality and the method's efficiency is validated against external benchmarks (MATH and LiveCodeBench v6) rather than derived from the bound, so the selection criterion is not a fitted input renamed as a prediction. Theorem 2's speedup ratio (Eq. 4) is derived from an explicit Pareto model and the external Razin et al. timescale law τ∝Var^{-1/3}; no experimental rollout numbers are injected into the formula, and the reported 2.3×/1.49× gains are measured, not generated by the formula. The only self-citation (Nguyen et al. 2024) appears in a related-work sentence and is not load-bearing; it is an external coreset paper and does not provide any premise used in the proofs or experiments. Appendix E.2's 7.9% false-negative measurement is an empirical robustness check, not a circular validation of the selection rule. No equation reduces to its own inputs, and no fitted constant is presented as a prediction.

Axiom & Free-Parameter Ledger

6 free parameters · 4 axioms · 0 invented entities

The central empirical claim does not depend on the theoretical model, but the theory (Theorem 2) and the efficiency justification rest on the Pareto assumption, the tau-proportional-to-Var^{-1/3} mapping, the binary-reward restriction, and the linear-rollout-time observation. No new entities are introduced.

free parameters (6)
  • selection ratio alpha = 0.5
    Hand-chosen hyperparameter controlling the fraction of prompts retained each round; fixed across all experiments; enters the speedup formula Eq. (4).
  • expansion ratio gamma = 2
    Hand-chosen multiplier for the rollout budget of retained prompts; with alpha=0.5 gives alpha*gamma=1 (budget-conserving).
  • refinement rounds T = 4
    Number of iterative allocation rounds; the theoretical speedup grows with T; chosen by hand.
  • initial rollout budget m0 = 2
    Number of rollouts per prompt in the first round; hand-chosen so the total budget matches GRPO's 8 rollouts per prompt.
  • Pareto shape k
    Introduced ad hoc for Theorem 2; the speedup ratio depends on k, and the qualitative conclusion requires k>1. No empirical fit or validation of the Pareto tail is reported.
  • Pareto threshold u_min
    Scale parameter of the assumed Pareto distribution; cancels in the budget-conserving speedup ratio, so it is not load-bearing for the headline numbers, but is part of the distributional assumption.
axioms (4)
  • ad hoc to paper Prompt-level reward variance u has an infinite-support Pareto upper tail with shape k>1 (f(u) = k u_min^k / u^{k+1} for u >= u_min).
    Assumed in Appendix D.2 to derive the closed-form speedup ratio; no empirical validation of the Pareto form is provided, and actual binary-reward variance is bounded above by 1, so the infinite tail is an idealization.
  • domain assumption Training timescale tau relates to expected reward variance as tau = C * E[Var]^{-1/3} (Eq. 23), treated as an equality rather than a lower bound.
    The paper cites Razin et al. (2025) for a lower bound and then writes the characteristic time as exactly this form, absorbing constants in C. The exact scaling and its extension to a mixture of prompts is assumed, not proved.
  • domain assumption Binary rewards r_i in {+1, -1} and KL-free GRPO (DAPO-style) govern the gradient-magnitude bound in Theorem 1.
    Restricts the theoretical analysis; many RLVR setups use continuous or partial rewards (e.g., the coding average test pass rate), so the exact identity S = G*sigma may not hold in general.
  • domain assumption Rollout generation time scales linearly with the number of rollouts and dominates training cost (Section 3.2, Figure 2 left).
    Empirical observation used to justify using rollout count as the efficiency metric; if overhead or non-linear scaling dominates, the 2.3x/1.49x claims do not translate to wall-clock speedups.

pith-pipeline@v1.3.0-alltime-deepseek · 18808 in / 21740 out tokens · 215378 ms · 2026-08-01T06:06:19.808672+00:00 · methodology

0 comments
read the original abstract

Reinforcement learning with verifiable rewards (RLVR) has emerged as a highly effective framework for improving LLM reasoning, with methods such as GRPO among its most successful instantiations. However, GRPO relies on repeated generation of long chain-of-thought rollouts. Training time scales with the number of rollouts, a large fraction of which are uninformative. Thus, GRPO is computationally expensive and unstable. To mitigate this, existing approaches either generate a larger pool of rollouts and filter the most informative prompts, or leverage historical signals for filtering at later stages of training. These strategies offer modest performance gains, but slow down the overall process. To address this, we propose VarIance Guided Online Rollout allocation (VIGOR) which instead of allocating a fixed rollout budget per example, begins with a small number of rollouts for all examples in a batch and iteratively allocates additional rollouts to those with the highest group reward variance until a fixed total rollout budget is reached. Theoretically, we show that under RLVR, reward variance controls the gradient magnitude, and derive VIGOR's closed-form speedup ratio over GRPO, which grows with refinement rounds under Pareto-distributed reward variance. Experiments on mathematical reasoning and coding tasks show that VIGOR reaches target accuracy with up to 2.3$\times$ fewer rollouts on math, reaches GRPO's final coding full pass rate with 1.49$\times$ fewer rollouts, and improves the coding average test pass rate by 3.4 points.

Figures

Figures reproduced from arXiv: 2607.22002 by Baharan Mirzasoleiman, Henry Liu, Heyang Jiang.

Figure 1
Figure 1. Figure 1: Left: VIGOR Coding results substantially outperform GRPO baseline on Live [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Empirical observations motivating VIGOR. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Overview of VIGOR. The method performs iterative rollout generation and [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Generation and verification time under different scaling factors, measured on [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Ablation and large-group analysis. Left: Ablation on prompt selection strategies. We compare our variance-based selection (VIGOR) with difficulty-based and random selection on Qwen2.5-3B under the same training setting. VIGOR consistently achieves the best performance, supporting our theoretical motivation. Right: Large-group comparison under n = 32, where VIGOR remains more rollout-efficient than GRPO and… view at source ↗
Figure 6
Figure 6. Figure 6: Rollout efficiency on the remaining math model settings. VIGOR is compared [PITH_FULL_IMAGE:figures/full_fig_p014_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Ablation on alternative rollout￾allocation schedules measured by validation accuracy against the number of rollouts con￾sumed. Every method uses the same total number of rollouts per gradient update step for a fair comparison. To study this trade-off, we compare VIGOR with three alternative designs. The first vari￾ant, T=2, m0=4, α=25%, γ=4, increases the number of rollouts generated in each iter￾ation whi… view at source ↗
Figure 8
Figure 8. Figure 8: Additional robustness results. Left: average test pass rate on LCBv6 with Qwen3- 8B. Middle: multi-seed training curves on MATH with Qwen2.5-1.5B, where each method is run across three random seeds (shaded regions denote variation across seeds) and VIGOR achieves an average training-efficiency speedup of 42.1% over GRPO. Right: distribution of the number of correct rollouts (out of 8 measurement rollouts) … view at source ↗
Figure 9
Figure 9. Figure 9: Effective sample ratio and emergent curriculum. [PITH_FULL_IMAGE:figures/full_fig_p020_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Cumulative distribution of rollout counts per prompt. Prompts with high selection frequency correspond to the model’s learning frontier. These prompts tend to lie at an intermediate difficulty level: they are not trivially solved across all rollouts, yet re￾main sufficiently interpretable for the model to produce responses that engage with the under￾lying mathematical structure. In contrast, low￾selection… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

85 extracted references · 27 linked inside Pith

  1. [1]

    AIME problems and solutions for 2024

    Art of Problem Solving . AIME problems and solutions for 2024. Online, 2024 a . URL https://artofproblemsolving.com/wiki/index.php/AIME_Problems_and_Solutions. Accessed: 2026-03-27

    2024

  2. [2]

    AMC problems and solutions for 2024

    Art of Problem Solving . AMC problems and solutions for 2024. Online, 2024 b . URL https://artofproblemsolving.com/wiki/index.php/AMC_Problems_and_Solutions. Accessed: 2026-03-27

    2024

  3. [3]

    Xrpo: Pushing the limits of grpo with targeted exploration and exploitation

    Udbhav Bamba, Minghao Fang, Yifan Yu, Haizhong Zheng, and Fan Lai. Xrpo: Pushing the limits of grpo with targeted exploration and exploitation. arXiv preprint arXiv:2510.06672, 2025

    Pith/arXiv arXiv 2025

  4. [4]

    Alpagasus: Training a better alpaca with fewer data

    Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, et al. Alpagasus: Training a better alpaca with fewer data. In The Twelfth International Conference on Learning Representations (ICLR), 2024

    2024

  5. [5]

    Christiano, Jan Leike, Tom B

    Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, volume 30, pp.\ 4299--4307. Curran Associates, Inc., 2017

    2017

  6. [8]

    Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402.14008, 2024

    Pith/arXiv arXiv 2024

  7. [9]

    Measuring mathematical problem solving with the MATH dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Advances in Neural Information Processing Systems, volume 34, 2021

    2021

  8. [11]

    Livecodebench: Holistic and contamination free evaluation of large language models for code

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. In The Thirteenth International Conference on Learning Representations (ICLR), 2025

    2025

  9. [12]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. arXiv preprint arXiv:2309.06180, 2023

    Pith/arXiv arXiv 2023

  10. [13]

    Math-verify: Math verification library

    Hynek Kydlicek. Math-verify: Math verification library. Software, 2024. URL https://github.com/huggingface/Math-Verify. Version 0.6.1

    2024

  11. [14]

    Miranda, Alisa Liu, Nouha Dziri, et al

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, et al. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124, 2024

    Pith/arXiv arXiv 2024

  12. [15]

    Le, Myeongho Jeon, Kim Vu, Viet Lai, and Eunho Yang

    Thanh-Long V. Le, Myeongho Jeon, Kim Vu, Viet Lai, and Eunho Yang. No prompt left behind: Exploiting zero-variance prompts in llm reinforcement learning via entropy-guided advantage shaping. In The Fourteenth International Conference on Learning Representations (ICLR), 2026

    2026

  13. [16]

    Solving quantitative reasoning problems with language models

    Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with language models. arXiv preprint arXiv:2206.14858, 2022

    Pith/arXiv arXiv 2022

  14. [17]

    Limr: Less is more for rl scaling

    Xuefeng Li, Haoyang Zou, and Pengfei Liu. Limr: Less is more for rl scaling. arXiv preprint arXiv:2502.11886, 2025

    Pith/arXiv arXiv 2025

  15. [18]

    Let's verify step by step

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.20050, 2023

    Pith/arXiv arXiv 2023

  16. [19]

    Understanding r1-zero-like training: A critical perspective

    Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025

    Pith/arXiv arXiv 2025

  17. [20]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2019

    Pith/arXiv arXiv 2019

  18. [21]

    Dynamics-predictive sampling for active rl finetuning of large reasoning models

    Yixiu Mao, Yun Qu, Qi Wang, Heming Zou, and Xiangyang Ji. Dynamics-predictive sampling for active rl finetuning of large reasoning models. In The Fourteenth International Conference on Learning Representations (ICLR), 2026

    2026

  19. [23]

    s1: Simple test-time scaling

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Cand\`es, and Tatsunori Hashimoto. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393, 2025

    Pith/arXiv arXiv 2025

  20. [24]

    Mini-batch coresets for memory-efficient language model training on data mixtures

    Dang Nguyen, Wenhan Yang, Rathul Anand, Yu Yang, and Baharan Mirzasoleiman. Mini-batch coresets for memory-efficient language model training on data mixtures. arXiv preprint arXiv:2407.19580, 2024

    Pith/arXiv arXiv 2024

  21. [25]

    Adaptive rollout allocation for online reinforcement learning with verifiable rewards

    Hieu Trung Nguyen, Bao Nguyen, Wenao Ma, Yuzhi Zhao, Ruifeng She, and Viet Anh Nguyen. Adaptive rollout allocation for online reinforcement learning with verifiable rewards. In The Fourteenth International Conference on Learning Representations (ICLR), 2026

    2026

  22. [26]

    Openai o1 system card

    OpenAI . Openai o1 system card. Technical report, 2024. URL https://cdn.openai.com/o1-system-card-20241205.pdf. Accessed: 2026-03-22

    2024

  23. [27]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155, 2022

    Pith/arXiv arXiv 2022

  24. [28]

    Lee, and Sanjeev Arora

    Noam Razin, Zixuan Wang, Hubert Strauss, Stanley Wei, Jason D. Lee, and Sanjeev Arora. What makes a reward model a good teacher? an optimization perspective. arXiv preprint arXiv:2503.15477, 2025

    arXiv 2025

  25. [29]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    Pith/arXiv arXiv 2017

  26. [30]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    Pith/arXiv arXiv 2024

  27. [31]

    Hybridflow: A flexible and efficient rlhf framework

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv:2409.19256, 2024

    Pith/arXiv arXiv 2024

  28. [32]

    Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. Learning to summarize with human feedback. arXiv preprint arXiv:2009.01325, 2020

    Pith/arXiv arXiv 2009

  29. [33]

    Zico Kolter

    Yixuan Even Xu, Yash Savani, Fei Fang, and J. Zico Kolter. Not all rollouts are useful: Down-sampling rollouts in llm reinforcement learning. arXiv preprint arXiv:2504.13818, 2025

    Pith/arXiv arXiv 2025

  30. [34]

    Qwen2.5 technical report

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2024 a

    Pith/arXiv arXiv 2024

  31. [35]

    Qwen2.5-math technical report: Toward mathematical expert model via self-improvement

    An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, et al. Qwen2.5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12122, 2024 b

    Pith/arXiv arXiv 2024

  32. [36]

    Qwen3 technical report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025

    Pith/arXiv arXiv 2025

  33. [37]

    Limo: Less is more for reasoning

    Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning. arXiv preprint arXiv:2502.03387, 2025. COLM 2025

    Pith/arXiv arXiv 2025

  34. [38]

    Dapo: An open-source llm reinforcement learning system at scale

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025

    Pith/arXiv arXiv 2025

  35. [39]

    Evaluating the performance of large language models on gaokao benchmark

    Xiaotian Zhang, Chunyang Li, Yi Zong, Zhengyu Ying, Liang He, and Xipeng Qiu. Evaluating the performance of large language models on gaokao benchmark. arXiv preprint arXiv:2305.12474, 2023

    Pith/arXiv arXiv 2023

  36. [40]

    Improving sampling efficiency in rlvr through adaptive rollout and response reuse

    Yuheng Zhang, Wenlin Yao, Changlong Yu, Yao Liu, Qingyu Yin, Bing Yin, Hyokun Yun, and Lihong Li. Improving sampling efficiency in rlvr through adaptive rollout and response reuse. arXiv preprint arXiv:2509.25808, 2025

    arXiv 2025

  37. [41]

    Prosperity before collapse: How far can off-policy rl reach with stale data on llms? arXiv preprint arXiv:2510.01161, 2025 a

    Haizhong Zheng, Jiawei Zhao, and Beidi Chen. Prosperity before collapse: How far can off-policy rl reach with stale data on llms? arXiv preprint arXiv:2510.01161, 2025 a

    arXiv 2025

  38. [42]

    Bartoldson, Bhavya Kailkhura, Fan Lai, Jiawei Zhao, and Beidi Chen

    Haizhong Zheng, Yang Zhou, Brian R. Bartoldson, Bhavya Kailkhura, Fan Lai, Jiawei Zhao, and Beidi Chen. Act only when it pays: Efficient reinforcement learning for llm reasoning via selective rollouts. In The Thirty-Ninth Conference on Neural Information Processing Systems (NeurIPS), 2025 b

    2025

  39. [43]

    Davir: Data selection via implicit reward for large language models

    Haotian Zhou, Tingkai Liu, Qianli Ma, Yufeng Zhang, Jianbo Yuan, Pengfei Liu, Yang You, and Hongxia Yang. Davir: Data selection via implicit reward for large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 9220--9237. Association for Computational Linguistics, 2025

    2025

  40. [44]

    Attention is All you Need , url =

    Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N and Kaiser, ukasz and Polosukhin, Illia , booktitle =. Attention is All you Need , url =

  41. [45]

    and Le, Quoc V

    Wei, Jason and Wang, Xuezhi and Schuurmans, Dale and Bosma, Maarten and Ichter, Brian and Xia, Fei and Chi, Ed H. and Le, Quoc V. and Zhou, Denny , journal =. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models , year =

  42. [46]

    and Leike, Jan and Brown, Tom B

    Christiano, Paul F. and Leike, Jan and Brown, Tom B. and Martic, Miljan and Legg, Shane and Amodei, Dario , booktitle =. Deep Reinforcement Learning from Human Preferences , volume =

  43. [47]

    and Lowe, Ryan and Voss, Chelsea and Radford, Alec and Amodei, Dario and Christiano, Paul F

    Stiennon, Nisan and Ouyang, Long and Wu, Jeffrey and Ziegler, Daniel M. and Lowe, Ryan and Voss, Chelsea and Radford, Alec and Amodei, Dario and Christiano, Paul F. , journal =. Learning to Summarize with Human Feedback , year =

  44. [48]

    Training Language Models to Follow Instructions with Human Feedback , year =

    Ouyang, Long and Wu, Jeffrey and Jiang, Xu and Almeida, Diogo and Wainwright, Carroll and Mishkin, Pamela and Zhang, Chong and Agarwal, Sandhini and Slama, Katarina and others , journal =. Training Language Models to Follow Instructions with Human Feedback , year =

  45. [49]

    OpenAI o1 System Card , year =

  46. [50]

    arXiv preprint arXiv:2503.01743 , title =

    Pith/arXiv arXiv

  47. [51]

    Proximal Policy Optimization Algorithms , year =

    Schulman, John and Wolski, Filip and Dhariwal, Prafulla and Radford, Alec and Klimov, Oleg , journal =. Proximal Policy Optimization Algorithms , year =

  48. [52]

    Let's Verify Step by Step , year =

    Lightman, Hunter and Kosaraju, Vineet and Burda, Yura and Edwards, Harri and Baker, Bowen and Lee, Teddy and Leike, Jan and Schulman, John and Sutskever, Ilya and Cobbe, Karl , journal =. Let's Verify Step by Step , year =

  49. [53]

    Decoupled Weight Decay Regularization , year =

    Loshchilov, Ilya and Hutter, Frank , journal =. Decoupled Weight Decay Regularization , year =

  50. [54]

    2024 , howpublished =

    Kydlicek, Hynek , title =. 2024 , howpublished =

    2024

  51. [55]

    and Liu, Alisa and Dziri, Nouha and others , journal =

    Lambert, Nathan and Morrison, Jacob and Pyatkin, Valentina and Huang, Shengyi and Ivison, Hamish and Brahman, Faeze and Miranda, Lester James V. and Liu, Alisa and Dziri, Nouha and others , journal =. Tulu 3: Pushing Frontiers in Open Language Model Post-Training , year =

  52. [56]

    Shao, Zhihong and Wang, Peiyi and Zhu, Qihao and Xu, Runxin and Song, Junxiao and Bi, Xiao and Zhang, Haowei and Zhang, Mingchuan and Li, Y. K. and Wu, Y. and Guo, Daya , journal =. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models , year =

  53. [57]

    and Zhang, Hao and Stoica, Ion , journal =

    Kwon, Woosuk and Li, Zhuohan and Zhuang, Siyuan and Sheng, Ying and Zheng, Lianmin and Yu, Cody Hao and Gonzalez, Joseph E. and Zhang, Hao and Stoica, Ion , journal =. Efficient Memory Management for Large Language Model Serving with PagedAttention , year =

  54. [58]

    HybridFlow: A Flexible and Efficient RLHF Framework , year =

    Sheng, Guangming and Zhang, Chi and Ye, Zilingfeng and Wu, Xibin and Zhang, Wang and Zhang, Ru and Peng, Yanghua and Lin, Haibin and Wu, Chuan , journal =. HybridFlow: A Flexible and Efficient RLHF Framework , year =

  55. [59]

    arXiv preprint arXiv:2501.12948 , title =

    Pith/arXiv arXiv

  56. [60]

    Qwen3 Technical Report , year =

    Yang, An and Li, Anfeng and Yang, Baosong and Zhang, Beichen and Hui, Binyuan and Zheng, Bo and Yu, Bowen and Gao, Chang and Huang, Chengen and others , journal =. Qwen3 Technical Report , year =

  57. [61]

    LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code , year =

    Jain, Naman and Han, King and Gu, Alex and Li, Wen-Ding and Yan, Fanjia and Zhang, Tianjun and Wang, Sida and Solar-Lezama, Armando and Sen, Koushik and Stoica, Ion , booktitle =. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code , year =

  58. [62]

    arXiv preprint arXiv:2601.20802 , title =

    H. arXiv preprint arXiv:2601.20802 , title =

    Pith/arXiv arXiv

  59. [63]

    Qwen2.5 Technical Report , year =

    Yang, An and Yang, Baosong and Zhang, Beichen and Hui, Binyuan and Zheng, Bo and Yu, Bowen and Li, Chengyuan and Liu, Dayiheng and Huang, Fei and others , journal =. Qwen2.5 Technical Report , year =

  60. [64]

    Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement , year =

    Yang, An and Zhang, Beichen and Hui, Binyuan and Gao, Bofei and Yu, Bowen and Li, Chengpeng and Liu, Dayiheng and Tu, Jianhong and Zhou, Jingren and others , journal =. Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement , year =

  61. [65]

    and Arora, Sanjeev , journal =

    Razin, Noam and Wang, Zixuan and Strauss, Hubert and Wei, Stanley and Lee, Jason D. and Arora, Sanjeev , journal =. What Makes a Reward Model a Good Teacher? An Optimization Perspective , year =

  62. [66]

    DAPO: An Open-Source LLM Reinforcement Learning System at Scale , year =

    Yu, Qiying and Zhang, Zheng and Zhu, Ruofei and Yuan, Yufeng and Zuo, Xiaochen and Yue, Yu and Dai, Weinan and Fan, Tiantian and Liu, Gaohong and others , journal =. DAPO: An Open-Source LLM Reinforcement Learning System at Scale , year =

  63. [67]

    arXiv preprint arXiv:2512.02556 , title =

    Pith/arXiv arXiv

  64. [68]

    Understanding R1-Zero-Like Training: A Critical Perspective , year =

    Liu, Zichen and Chen, Changyu and Li, Wenjun and Qi, Penghui and Pang, Tianyu and Du, Chao and Lee, Wee Sun and Lin, Min , journal =. Understanding R1-Zero-Like Training: A Critical Perspective , year =

  65. [69]

    On the Design of KL-Regularized Policy Gradient Algorithms for LLM Reasoning , year =

    Zhang, Yifan and Liu, Yifeng and Yuan, Huizhuo and Yuan, Yang and Gu, Quanquan and Yao, Andrew Chi-Chih , journal =. On the Design of KL-Regularized Policy Gradient Algorithms for LLM Reasoning , year =

  66. [70]

    DCPO: Dynamic Clipping Policy Optimization , year =

    Yang, Shihui and Dou, Chengfeng and Guo, Peidong and Lu, Kai and Ju, Qiang and Deng, Fei and Xin, Rihui , journal =. DCPO: Dynamic Clipping Policy Optimization , year =

  67. [71]

    LIMR: Less is More for RL Scaling , year =

    Li, Xuefeng and Zou, Haoyang and Liu, Pengfei , journal =. LIMR: Less is More for RL Scaling , year =

  68. [72]

    DavIR: Data Selection via Implicit Reward for Large Language Models , year =

    Zhou, Haotian and Liu, Tingkai and Ma, Qianli and Zhang, Yufeng and Yuan, Jianbo and Liu, Pengfei and You, Yang and Yang, Hongxia , booktitle =. DavIR: Data Selection via Implicit Reward for Large Language Models , year =

  69. [73]

    AlpaGasus: Training a Better Alpaca with Fewer Data , year =

    Chen, Lichang and Li, Shiyang and Yan, Jun and Wang, Hai and Gunaratna, Kalpa and Yadav, Vikas and Tang, Zheng and Srinivasan, Vijay and Zhou, Tianyi and others , booktitle =. AlpaGasus: Training a Better Alpaca with Fewer Data , year =

  70. [74]

    s1: Simple test-time scaling , year =

    Muennighoff, Niklas and Yang, Zitong and Shi, Weijia and Li, Xiang Lisa and Fei-Fei, Li and Hajishirzi, Hannaneh and Zettlemoyer, Luke and Liang, Percy and Cand\`es, Emmanuel and Hashimoto, Tatsunori , journal =. s1: Simple test-time scaling , year =

  71. [75]

    LIMO: Less is More for Reasoning , year =

    Ye, Yixin and Huang, Zhen and Xiao, Yang and Chern, Ethan and Xia, Shijie and Liu, Pengfei , journal =. LIMO: Less is More for Reasoning , year =

  72. [76]

    and Kailkhura, Bhavya and Lai, Fan and Zhao, Jiawei and Chen, Beidi , booktitle =

    Zheng, Haizhong and Zhou, Yang and Bartoldson, Brian R. and Kailkhura, Bhavya and Lai, Fan and Zhao, Jiawei and Chen, Beidi , booktitle =. Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts , year =

  73. [77]

    Dynamics-Predictive Sampling for Active RL Finetuning of Large Reasoning Models , year =

    Mao, Yixiu and Qu, Yun and Wang, Qi and Zou, Heming and Ji, Xiangyang , booktitle =. Dynamics-Predictive Sampling for Active RL Finetuning of Large Reasoning Models , year =

  74. [78]

    and Jeon, Myeongho and Vu, Kim and Lai, Viet and Yang, Eunho , booktitle =

    Le, Thanh-Long V. and Jeon, Myeongho and Vu, Kim and Lai, Viet and Yang, Eunho , booktitle =. No Prompt Left Behind: Exploiting Zero-Variance Prompts in LLM Reinforcement Learning via Entropy-Guided Advantage Shaping , year =

  75. [79]

    Zico , journal =

    Xu, Yixuan Even and Savani, Yash and Fang, Fei and Kolter, J. Zico , journal =. Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning , year =

  76. [80]

    Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs? , year =

    Zheng, Haizhong and Zhao, Jiawei and Chen, Beidi , journal =. Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs? , year =

  77. [81]

    Adaptive Rollout Allocation for Online Reinforcement Learning with Verifiable Rewards , year =

    Nguyen, Hieu Trung and Nguyen, Bao and Ma, Wenao and Zhao, Yuzhi and She, Ruifeng and Nguyen, Viet Anh , booktitle =. Adaptive Rollout Allocation for Online Reinforcement Learning with Verifiable Rewards , year =

  78. [82]

    Improving Sampling Efficiency in RLVR through Adaptive Rollout and Response Reuse , year =

    Zhang, Yuheng and Yao, Wenlin and Yu, Changlong and Liu, Yao and Yin, Qingyu and Yin, Bing and Yun, Hyokun and Li, Lihong , journal =. Improving Sampling Efficiency in RLVR through Adaptive Rollout and Response Reuse , year =

  79. [83]

    XRPO: Pushing the Limits of GRPO with Targeted Exploration and Exploitation , year =

    Bamba, Udbhav and Fang, Minghao and Yu, Yifan and Zheng, Haizhong and Lai, Fan , journal =. XRPO: Pushing the Limits of GRPO with Targeted Exploration and Exploitation , year =

  80. [84]

    Mini-batch Coresets for Memory-efficient Language Model Training on Data Mixtures , year =

    Nguyen, Dang and Yang, Wenhan and Anand, Rathul and Yang, Yu and Mirzasoleiman, Baharan , journal =. Mini-batch Coresets for Memory-efficient Language Model Training on Data Mixtures , year =

Showing first 80 references.