Pith. sign in

REVIEW 3 major objections 4 minor 32 references

The paper claims most rollout compute in RLVR is wasted on already-decided saturated groups, and that SARA — a sequential allocator using each prompt's own early rollouts — recovers dynamic-sampling accuracy at 22–67% fewer rollouts.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 00:21 UTC pith:FLRYYDQF

load-bearing objection Genuinely new sequential-allocation idea with plausible empirical gains, but Theorem 2's savings proof has a real denominator bug that undercuts the advertised guarantee. the 3 major comments →

arxiv 2607.26253 v1 pith:FLRYYDQF submitted 2026-07-28 cs.LG

Early Verdicts, Better Budgets: Sequential Adaptive Rollout Allocation for Compute-Efficient RLVR

Pixel Nomand , Elena Voss , Marcus Hale , Sofia Reyes This is my paper
classification cs.LG MSC 62L1068T05
keywords reinforcement learning with verifiable rewardsrollout allocationsequential analysisoptimal stoppingGRPOdynamic samplingBeta-Bernoulli modelearly stopping
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that most of the rollout budget in RLVR training is wasted on prompts whose groups are already saturated — all correct or all incorrect — and that the verdict is usually reached within the first few rollouts. It proposes SARA, an allocator that probes each prompt with a few rollouts, updates a Beta belief over its success rate, and stops early: commit a group once it is mixed, abandon it once it is almost surely saturated, and reallocate the freed budget to fresh prompts. SARA needs no extra prediction rollouts, and the authors prove bounds on abandonment reliability, expected savings, and fixed-budget yield. Empirically it matches a dynamic-sampling oracle's accuracy at 22% fewer rollouts, and composing it with predictive prompt selection beats the oracle by a small margin at 67% fewer rollouts. The contribution is a budget axis orthogonal to prompt selection: deciding in-sample, from a group's own rollouts, rather than predicting before or paying after.

Core claim

The central claim is that a prompt's training value — whether its rollout group will be effective (mixed successes) or saturated — is statistically decided long before the full group is generated, so collecting remaining rollouts is pure waste. SARA operationalizes this: for each prompt it keeps a Beta posterior over the success rate, evaluates a closed-form posterior-predictive probability peff that the completed group will be effective, and applies a two-threshold sequential probability-ratio-style rule. Effectiveness is absorbing (once mixed, always mixed) and along an all-same prefix peff decreases monotonically, so every prompt has a well-defined stopping time. The authors prove that ab

What carries the argument

The central object is the closed-form posterior-predictive effectiveness probability peff(n,s) = 1 − B(α0, β0+n+r)/B(α0, β0+n) (with symmetric all-pass form), computed from a Beta(α0,β0) prior and observed successes s in n rollouts with r=k−n remaining. It is exact for the event 'completed group of size k is mixed', and it defines the stopping rule: commit when mixed, abandon when peff < τlow, otherwise continue. The two-threshold rule is a one-sided sequential probability ratio test of 'saturated' against 'mixed', with zero type-I error (a committed group is always effective) and type-II error bounded by τlow. The same predictor doubles as a priority index for reallocating abandoned budget

Load-bearing premise

The entire closed-form predictor and stopping boundary assume that rollouts within a group are conditionally independent and identically distributed Bernoulli draws given a per-prompt success rate; if real rollouts are correlated (shared prefixes, multi-turn state) or rewards are continuous, the early-abandonment decisions that produce the savings can be miscalibrated — a limitation the paper states explicitly in App. G.

What would settle it

Run the same protocol on a task where within-group rollouts share a long prefix (e.g. multi-turn planning) and measure the empirical probability that an abandoned group would have been effective. If it exceeds τlow by a wide margin, the i.i.d. assumption is violated and the claimed savings are not reliable. Alternatively, on a dataset with non-monotone rewards where effectiveness is not absorbing, check whether the fraction of groups decided after n rollouts still reaches 1 as predicted.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • SARA matches the accuracy of dynamic sampling at 22% fewer rollouts on math reasoning and planning with 1.5B and 3B models on a single GPU.
  • Composing SARA with predictive prompt selection yields the best pass@1, slightly above the dynamic-sampling oracle, at 67% fewer rollouts — near-uniform cost.
  • SARA replaces oversample-and-filter: it assembles B effective groups at roughly B/Φ expected rollouts instead of Bk/Φ, and the savings grow with group size k.
  • Because abandoned all-fail traces are the longest, token savings (23%) exceed rollout savings, cutting GPU-hours roughly in line.
  • SARA is algorithm-agnostic: it improves PPO, RLOO, and Reinforce++ as well as GRPO when it replaces uniform rollout collection at matched effective-batch size.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same early-decidability argument should transfer to agentic and tree-based rollouts if the predictor is conditioned on prefix history rather than a single success rate — the authors flag this as future work, and it is the most direct path to testing the method beyond i.i.d. settings.
  • If the conditional-i.i.d. assumption fails in practice, the closed-form peff will be miscalibrated; a concrete fix is to learn a per-prefix effectiveness model and re-derive the stopping boundary, which would determine how much of the claimed savings survive correlated rollouts.
  • SARA's Beta posterior could be reused as a difficulty score for test-time compute allocation: an early-abandoned prompt is likely easy or hard, and the posterior gives a per-prompt uncertainty that a test-time sampler could query.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes SARA, a sequential adaptive rollout allocator for RLVR/GRPO training. It treats per-step rollout collection as an optimal-stopping problem over a stream of prompts: a Beta posterior over each prompt's success rate is updated from the group's own rollouts, and a closed-form posterior-predictive probability peff of eventual group effectiveness drives a two-threshold rule that commits mixed groups, abandons likely-saturated groups after a short probe, and reallocates the freed budget to fresh prompts. The authors claim proofs of abandonment reliability, expected rollout savings relative to dynamic sampling, fixed-budget yield dominance over uniform allocation, and a lower bound connecting effective-group yield to the GRPO gradient norm. Experiments on 1.5B/3B models for math and planning report that SARA matches dynamic-predictive sampling (DPS) while using 22% fewer rollouts than dynamic sampling (DS), and that SARA composed with DPS exceeds DS accuracy at 67% fewer rollouts.

Significance. If the theoretical claims hold, the core idea is useful and well motivated: a group's effectiveness is often decided before the full k rollouts are spent, and the closed-form Beta-Binomial predictor is elegant and parameter-free. The composition of an in-sample sequential allocator with predictive prompt selection is a sensible and practical contribution, and the reported empirical gains are substantial. The paper is clearly written and the empirical protocol is detailed. However, the advertised expected-savings theorem is not proven as written; this is a load-bearing gap because the abstract and Section 3.4 explicitly list 'expected rollout savings' among the proven contributions.

major comments (3)
  1. [App. C.4, Eq. (6); Theorem 2 in Section 3.4] The derivation of cost_SARA uses B/Phi for both schemes. Under SARA, a prompt is evaluated only up to N = min(Tmix, n_a), and it yields an effective group only when Tmix <= n_a. The probability of this is Psi = E[1 - gamma^{n_a} - (1-gamma)^{n_a}], which is strictly less than Phi = E[1 - gamma^k - (1-gamma)^k] whenever tau_low > 0. Hence SARA must evaluate B/Psi prompts in expectation, not B/Phi, and Eq. (6) should be corrected (e.g., cost_SARA = (B/Psi) E[N], or a per-effective-group comparison). The claimed strict inequality versus DS = (B/Phi) k is therefore not established by the current proof. Because 'expected rollout savings' is advertised as a proven result, this needs a corrected proof or a weakened statement.
  2. [Section 3.4 Theorem 2 / App. C.4] The theorem states n*(gamma) = O(log(1/tau_low) / log(1/max(gamma,1-gamma))) for saturated prompts. Under the default uniform prior used in all experiments, the abandonment boundary is n_a = k - floor(tau_low (k+1)), which is linear in k, and Eq. (5) gives E[N(gamma)] <= n_a. The logarithmic bound appears only in a parenthetical about an informative prior, which is not the default. The theorem as stated overclaims the strength of the guarantee for the configuration actually evaluated.
  3. [App. C.5, Theorem 3] The exchange argument says reallocated budget starts fresh prompts that are 'effective with probability Phi.' But under SARA a fresh prompt is effective only if it mixes before n_a, i.e., with probability Psi < Phi. The proof does not account for this reduction in per-prompt effectiveness and therefore does not establish the stated strict fixed-budget yield dominance. The conclusion may still be true, but a quantitative per-rollout yield comparison is needed.
minor comments (4)
  1. [Section 3.4 Theorem 4 / App. C.6] The main-text statement of Theorem 4 omits assumptions (i) and (ii) that appear in App. C.6. As written, 'Consequently' is not a logical consequence; the assumptions should be stated in the theorem.
  2. [Tables 1 and 3] There are small inconsistencies between Table 1 and Table 3: e.g., DPS average 53.2 vs 53.1, SARA 53.1 vs 53.3, SARA+DPS 54.8 vs 54.9. Please reconcile or state that the numbers come from different runs/roundings.
  3. [Eq. (2) and surrounding text] The notation peff vs p_eff is used inconsistently. Please unify the symbol and the typesetting of the closed-form predictor.
  4. [Figure 6c] The labels in Figure 6c appear clipped/overlapping, especially the y-axis 'effective wrongly abandoned (%)'. A cleaner layout would help readability.

Circularity Check

0 steps flagged

No significant circularity: SARA's predictor is a parameter-free Bayesian quantity and results are measured against external benchmarks; the savings guarantee has a non-circular proof gap.

full rationale

The paper's central derivation—the peff predictor and stopping rule—is not fitted to the reported accuracies. Eq. 2 is a closed-form posterior predictive from a Beta-Bernoulli model with a stated uniform prior; it is not defined in terms of the outcome it predicts, and the empirical pass@1 numbers come from external benchmarks (AIME, AMC, MATH500, etc.), so they are not re-statements of the model's assumptions. There are no load-bearing self-citations: the theoretical apparatus cites Wald, Robbins, Gittins, Thompson, all external and independent. The headline rollout-savings figure is contingent on the tunable threshold tau_low=0.45, which is a hyperparameter-sensitivity issue, not a circular reduction. The one serious derivation problem is non-circular: App. C.4 Eq. 6 uses the dynamic-sampling effectiveness probability Phi for SARA ('both schemes must evaluate, in expectation, B/Phi prompts'), whereas SARA's early abandonment makes its per-prompt effectiveness probability Psi < Phi, so the expected-savings theorem is not proven as written. This is a mathematical correctness gap, not a case of a prediction being equivalent to its inputs by construction. Accordingly, no circular step meets the bar of quote-and-reduction.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

No invented physical entities. The central derivation rests on a Beta-Bernoulli i.i.d. model, a hand-set threshold/probe size, and an auxiliary nonnegative cross-alignment assumption in Thm 4. tau_low is the main free knob; the abstract's savings percentages are contingent on it.

free parameters (5)
  • tau_low = 0.45
    Abandon threshold in Eq. (3); the single knob controlling the savings/loss trade-off. Chosen as default from the sweep in Fig. 6a; headline 22% savings and 8.6% abandonment loss depend on this choice.
  • n0 = 2
    Initial probe size in Algorithm 1; swept over {1,...,4} (Fig. 6b), default 2.
  • alpha0, beta0 = 1,1
    Uniform Beta prior; standard Bayesian default, not fitted to data, but a modeling choice used by the closed-form predictor.
  • commit_min = k (default 8)
    Minimum group size before commit; default equals k (fixed-size groups); ablation uses m=4 (Fig. 6c) and changes the cost profile.
  • safety budget cap = 6Bk
    Implementation cap to bound worst-case spend; not part of the theory.
axioms (5)
  • domain assumption Within a prompt, rollouts are conditionally i.i.d. Bernoulli(gamma) given success rate gamma.
    Assumed in App. C intro and used to derive the Beta-Binomial predictor (Proposition 1) and all stopping-time calculations; acknowledged as a limitation in App. G.
  • domain assumption Rewards are binary verifiable correctness.
    The predictor models success/failure only; continuous/format rewards are binarized (App. E, F).
  • domain assumption A Beta prior with fixed parameters on each prompt's success rate.
    Bayesian model choice; uniform by default, warm-start optional; not derived from data.
  • ad hoc to paper Assumption (ii) in Thm 4: average cross-alignment of gradients from different effective groups is nonnegative.
    Needed to convert E||grad J||^2 = sum group norm squares + cross terms into a lower bound cE[#effective]; not verified empirically.
  • standard math Saturated groups have zero advantage and hence zero gradient.
    Definition 1 + Eq. (1): zero reward variance yields A_i = 0; degenerate group dropped.

pith-pipeline@v1.3.0-alltime-deepseek · 17452 in / 18346 out tokens · 175753 ms · 2026-08-01T00:21:38.455710+00:00 · methodology

0 comments
read the original abstract

Reinforcement learning with verifiable rewards (RLVR) is bottlenecked by rollout generation, yet many sampled prompts produce saturated groups (all responses correct or all incorrect) whose zero reward variance yields no policy-gradient signal. Existing remedies either oversample a larger candidate pool and discard saturated prompts (dynamic sampling), paying heavy extra rollouts, or predict prompt difficulty before sampling, which is fragile under a shifting policy. We observe that a group's effectiveness is usually decided early, within the first few of its rollouts, so spending a full group on an already-decided prompt is wasteful. We cast per-step rollout collection as a budget-constrained sequential allocation (optimal stopping) problem and introduce SARA (Sequential Adaptive Rollout Allocation). SARA maintains a Beta posterior over each prompt's success rate, evaluates a closed-form predictor of group effectiveness, and applies a two-threshold, SPRT-style rule that commits effective groups, abandons saturated ones after a short probe, and reallocates the freed budget to fresh prompts, without any extra prediction rollouts. We prove abandonment reliability, expected rollout savings, fixed-budget yield dominance, and a link between effective-group yield and the GRPO gradient norm. On mathematical reasoning and planning with 1.5B/3B models on a single GPU, SARA matches DPS (both below the DS oracle) while using 22% fewer rollouts than DS; composing SARA with DPS yields the best accuracy, slightly above DS, at 67% fewer rollouts (near-uniform cost).

Figures

Figures reproduced from arXiv: 2607.26253 by Elena Voss, Marcus Hale, Pixel Nomand, Sofia Reyes.

Figure 1
Figure 1. Figure 1: Sequential allocation spends rollouts where verdicts are contrastive. Left: fixed-k allocation (uniform sampling, or dynamic sampling after filtering) generates a full group of k rollouts for every prompt; saturated groups (all-correct or all-incorrect, faded) have zero reward variance and contribute no GRPO gradient. Right: SARA abandons a group as soon as its outcome is statistically decided and realloca… view at source ↗
Figure 2
Figure 2. Figure 2: Saturated groups dominate, and SARA keeps the batch effective. (a) Distribution of per-group success counts under uniform sampling (k=8): most groups sit at s=0 or s=k (zero gradient). (b) Fraction of the training batch that is effective over training: SARA (and the dynamic￾sampling oracle) deliver near-fully effective batches; predictive selection recovers most of it but degrades as the policy shifts; uni… view at source ↗
Figure 3
Figure 3. Figure 3: Mechanism of SARA. (a) Fraction of groups whose final effectiveness label is already decided after n observed rollouts (committed-effective + confidently-abandoned); most verdicts are reached well before n=k, and they agree with the oracle label. (b) The closed-form predictor p eff(n, s) of Eq. equation 2 with the abandon region (p eff < τlow) outlined. (c) Effective groups collected versus rollouts spent:… view at source ↗
Figure 4
Figure 4. Figure 4: SARA inside one GRPO step. Prompts are probed in batched rounds; a Beta posterior and the closed-form predictor p eff route each group to COMMIT/CONTINUE/ABANDON; abandoned budget is reallocated to fresh prompts, and only effective groups reach the update. No auxiliary prediction rollouts are used. Two-threshold sequential rule. Given p eff(n, s), SARA applies a sequential test reminiscent of Wald’s SPRT (… view at source ↗
Figure 5
Figure 5. Figure 5: Accuracy and cost over training. (a, b) Pass@1 vs. optimizer step on MATH (1.5B) and Countdown (3B); every method runs for the same number of steps. (c) MATH vs. cumulative rollouts: SARA+DPS reaches above-DS accuracy with 67% fewer rollouts. down (Pan et al., 2025) splits with GRPO on verl (Sheng et al., 2024), and evaluate pass@1 (mean over 16 samples) on AIME24, AMC23, MATH500 (Lightman et al., 2023), M… view at source ↗
Figure 6
Figure 6. Figure 6: Ablations. (a) abandon threshold τlow; (b) probe size n0; (c) component ablation (effective groups under a fixed budget equal to DS’s per-step cost); (d) savings vs. DS grow with group size k [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Cost to assemble a clean batch. (a) SARA uses ∼22%/23% fewer rollouts/tokens than the DS oracle for an equally effective batch of B groups. (b) All-fail (hard) rollouts are the longest, so abandoning them after a short probe saves disproportionate tokens. RL configuration. GRPO on verl (Sheng et al., 2024): group size k=8, effective batch B=64, learning rate 1×10−6 , KL coefficient 1×10−3 , clip ϵ=0.2, max… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

32 extracted references · 23 linked inside Pith

  1. [1]

    L1: Controlling how long a reasoning model thinks with reinforcement learning.arXiv preprint arXiv:2503.04697,

    Pranjal Aggarwal and Sean Welleck. L1: Controlling how long a reasoning model thinks with reinforcement learning.arXiv preprint arXiv:2503.04697,

    Pith/arXiv arXiv

  2. [4]

    Defaults and history warm-start.Defaults: B=64, k=8, n0=2, τlow=0.45, commit_min=k, prior Beta(1,1) , safety budget cap 6Bk

    is reported as an ablation in Figure 6c and further reduces rollouts at the cost of variable group sizes. Defaults and history warm-start.Defaults: B=64, k=8, n0=2, τlow=0.45, commit_min=k, prior Beta(1,1) , safety budget cap 6Bk. The optional warm-start carries each prompt’s posterior across steps with an exponential decayλ (so a prompt seen to be satura...

    2026

  3. [8]

    REINFORCE++: Stabilizing critic-free policy optimization with global advantage normal- ization.arXiv preprint arXiv:2501.03262,

    9 Preprint Jian Hu. REINFORCE++: Stabilizing critic-free policy optimization with global advantage normal- ization.arXiv preprint arXiv:2501.03262,

    Pith/arXiv arXiv

  4. [9]

    Openai o1 system card.arXiv preprint arXiv:2412.16720,

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card.arXiv preprint arXiv:2412.16720,

    Pith/arXiv arXiv

  5. [11]

    Let’s verify step by step.arXiv preprint arXiv:2305.20050,

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step.arXiv preprint arXiv:2305.20050,

    Pith/arXiv arXiv

  6. [13]

    Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models

    Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong. Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models. arXiv preprint arXiv:2505.24864, 2025a. Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like traini...

    Pith/arXiv arXiv

  7. [16]

    Hybridflow: A flexible and efficient RLHF framework.arXiv preprint arXiv:2409.19256,

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient RLHF framework.arXiv preprint arXiv:2409.19256,

    Pith/arXiv arXiv

  8. [17]

    Stop overthinking: A survey on efficient reasoning for large language models.arXiv preprint arXiv:2503.16419,

    Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen Zhong, Hanjie Chen, et al. Stop overthinking: A survey on efficient reasoning for large language models.arXiv preprint arXiv:2503.16419,

    Pith/arXiv arXiv

  9. [18]

    Kimi k1.5: Scaling reinforcement learning with LLMs.arXiv preprint arXiv:2501.12599,

    10 Preprint Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1.5: Scaling reinforcement learning with LLMs.arXiv preprint arXiv:2501.12599,

    Pith/arXiv arXiv

  10. [20]

    Qwen2.5 technical report.arXiv preprint arXiv:2412.15115,

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2.5 technical report.arXiv preprint arXiv:2412.15115,

    Pith/arXiv arXiv

  11. [21]

    Limo: Less is more for reasoning.arXiv preprint arXiv:2502.03387,

    Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning.arXiv preprint arXiv:2502.03387,

    Pith/arXiv arXiv

  12. [22]

    Dapo: An open-source LLM reinforcement learning system at scale.arXiv preprint arXiv:2503.14476,

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source LLM reinforcement learning system at scale.arXiv preprint arXiv:2503.14476,

    Pith/arXiv arXiv

  13. [23]

    Srpo: A cross-domain implementation of large-scale reinforcement learning on LLM.arXiv preprint arXiv:2504.14286,

    Xiaojiang Zhang, Jinghui Wang, Zifei Cheng, Wenhao Zhuang, Zheng Lin, Minglei Zhang, Shaojie Wang, Yinghan Cui, Chao Wang, Junyi Peng, et al. Srpo: A cross-domain implementation of large-scale reinforcement learning on LLM.arXiv preprint arXiv:2504.14286,

    Pith/arXiv arXiv

  14. [24]

    Act only when it pays: Efficient reinforcement learning for LLM reasoning via selective rollouts.arXiv preprint arXiv:2506.02177,

    Haizhong Zheng, Yang Zhou, Brian R Bartoldson, Bhavya Kailkhura, Fan Lai, Jiawei Zhao, and Beidi Chen. Act only when it pays: Efficient reinforcement learning for LLM reasoning via selective rollouts.arXiv preprint arXiv:2506.02177,

    Pith/arXiv arXiv

  15. [25]

    Trace: A unified rollout budget allocation framework for efficient agentic reinforcement learning.arXiv preprint arXiv:2606.11119,

    Heming Zou, Qi Wang, Yun Qu, Yuhang Jiang, Lizhou Cai, Yixiu Mao, Ru Peng, Xin Xu, Weijie Liu, Kai Yang, Saiyong Yang, and Xiangyang Ji. Trace: A unified rollout budget allocation framework for efficient agentic reinforcement learning.arXiv preprint arXiv:2606.11119,

    Pith/arXiv arXiv

  16. [27]

    value network with a group-normalized advantage; many variants tune stability, bias, and length (Liu et al., 2025b; Hu, 2025; Yu et al., 2025; Liu et al., 2025a). The recurring principle is that thewithin-group reward varianceis the source of signal (exactly what saturated groups lack), which makes assembling effective groups the central efficiency lever ...

    2025

  17. [28]

    (ii) Rollout allocationsets how many rollouts each prompt or trajectory prefix receives (Zou et al.,

    and online difficulty filtering (Bae et al., 2025)) oversample a candidate pool, generate full groups, and discard saturated ones; andpredict-then-select methods forecast difficultybeforesampling (Mao et al., 2026; Qu et al., 2026; Chen et al., 2025b). (ii) Rollout allocationsets how many rollouts each prompt or trajectory prefix receives (Zou et al.,

    2025

  18. [29]

    These all commit budgetbeforea group’s own rollouts are observed (evaluate-then-filter even pays full groups for the prompts it discards)

    and prunes low-value generations (Zheng et al., 2025; Lin et al., 2025). These all commit budgetbeforea group’s own rollouts are observed (evaluate-then-filter even pays full groups for the prompts it discards). SARA instead decidesduringa group’s own rollouts with an in-sample sequential verdict: it needs no auxiliary evaluation passes, is robust to the ...

    2025

  19. [30]

    gives the optimal index policy in the discounted relaxation; SARA’s priority by peff is its one-step (myopic) approximation, and posterior-sampling priority (Thompson, 1933; Russo & Van Roy,

    1933

  20. [32]

    effective

    training split and a subset of Countdown (Pan et al., 2025). Math evaluation reports pass@1 (mean over 16 samples, temperature 1.0) on AIME24, AMC23, MATH500 (Lightman et al., 2023), Minerva (Lewkowycz et al., 2022), and OlympiadBench (He et al., 2024). Planning reports pass@1 on the in-distribution CD-34 and the harder CD-4 split. Rewards are binary veri...

    2025

  21. [1945]

    Reinforcement learning for reasoning in large language models with one training example.arXiv preprint arXiv:2504.20571,

    Yiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren, Liyuan Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, et al. Reinforcement learning for reasoning in large language models with one training example.arXiv preprint arXiv:2504.20571,

    Pith/arXiv arXiv

  22. [1972]

    Towards reasoning era: A survey of long chain-of- thought for reasoning large language models.arXiv preprint arXiv:2503.09567, 2025a

    Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng, Jiannan Guan, Peng Wang, Mengkang Hu, Yuhang Zhou, Te Gao, and Wanxiang Che. Towards reasoning era: A survey of long chain-of- thought for reasoning large language models.arXiv preprint arXiv:2503.09567, 2025a. Xiaoyin Chen, Jiarui Lu, Minsu Kim, Dinghuai Zhang, Jian Tang, Alexandre Piché, Nicolas Gontier,...

    Pith/arXiv arXiv

  23. [1979]

    Deepseek-r1: Incentivizing reasoning capability in LLMs via reinforcement learning.arXiv preprint arXiv:2501.12948,

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in LLMs via reinforcement learning.arXiv preprint arXiv:2501.12948,

    Pith/arXiv arXiv

  24. [2002]

    Online difficulty filtering for reasoning oriented reinforcement learning.arXiv preprint arXiv:2504.03380,

    Sanghwan Bae, Jiwoo Hong, Min Young Lee, Hanbyul Kim, JeongYeon Nam, and Donghyun Kwak. Online difficulty filtering for reasoning oriented reinforcement learning.arXiv preprint arXiv:2504.03380,

    arXiv

  25. [2014]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

    Pith/arXiv arXiv

  26. [2017]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y . K. Li, Y . Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models.arXiv preprint arXiv:2402.03300,

    Pith/arXiv arXiv

  27. [2021]

    Thinkprune: Pruning long chain-of-thought of LLMs via reinforcement learning.arXiv preprint arXiv:2504.01296,

    Bairu Hou, Yang Zhang, Jiabao Ji, Yujian Liu, Kaizhi Qian, Jacob Andreas, and Shiyu Chang. Thinkprune: Pruning long chain-of-thought of LLMs via reinforcement learning.arXiv preprint arXiv:2504.01296,

    Pith/arXiv arXiv

  28. [2022]

    Limr: Less is more for RL scaling.arXiv preprint arXiv:2502.11886,

    Xuefeng Li, Haoyang Zou, and Pengfei Liu. Limr: Less is more for RL scaling.arXiv preprint arXiv:2502.11886,

    Pith/arXiv arXiv

  29. [2023]

    Cppo: Accelerating the training of group relative policy optimization-based reasoning models.arXiv preprint arXiv:2503.22342,

    Zhihang Lin, Mingbao Lin, Yuan Xie, and Rongrong Ji. Cppo: Accelerating the training of group relative policy optimization-based reasoning models.arXiv preprint arXiv:2503.22342,

    arXiv

  30. [2024]

    Measuring mathematical problem solving with the MATH dataset.arXiv preprint arXiv:2103.03874,

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset.arXiv preprint arXiv:2103.03874,

    Pith/arXiv arXiv

  31. [2025]

    Olympiadbench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems.arXiv preprint arXiv:2402.14008,

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems.arXiv preprint arXiv:2402.14008,

    Pith/arXiv arXiv

  32. [2026]

    No LLM was used to generate experimental results, proofs, or claims; all theoretical statements and their proofs (App

    11 Preprint A USE OFLARGELANGUAGEMODELS Large language models were used solely as general-purpose assistive tools during manuscript prepa- ration: polishing wording, checking LaTeX, and suggesting references that the authors subsequently verified against primary sources. No LLM was used to generate experimental results, proofs, or claims; all theoretical ...

    2025