Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Effective Reinforcement Learning for Reasoning in Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A new RL algorithm for reasoning models, DASH, claims an 83% training-time reduction over a standard GRPO implementation while keeping accuracy essentially unchanged.

desk verdict Useful batch-size insight for RL on small LMs, but the 83% speedup and PPO conclusions need error bars and a fairer GRPO baseline before I'd trust them. read the letter →

arxiv 2505.17218 v1 pith:3PD4QSV2 submitted 2025-05-22 cs.AI

Lianghuan Huang , Shuo Li , Sagnik Anupam , Insup Lee , Osbert Bastani This is my paper
classification cs.AI
keywords reinforcementlearninglanguagemodelreasoningGRPOpreemptivesamplinggradientfilteringpolicyKLdivergencetrainingefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the usual RL machinery built for robotics is mismatched to language-model reasoning, both in accuracy and in compute. It reports three empirical findings on 0.5B-3B Qwen2.5 base models: on-policy reinforcement learning clearly improves MATH and GSM8K accuracy while supervised fine-tuning on expert traces does not; PPO-style multi-step gradient updates add instability rather than stability; and removing KL divergence against the base model gives shorter generations and usually higher accuracy. The paper's central contribution is DASH, which combines preemptive sampling (generate a large inference batch, then accumulate gradients in small increments) with gradient filtering (drop samples with small advantage estimates). On MATH-500 with Qwen2.5-0.5B, DASH completes in 6.6 hours versus 38.9 hours for a standard GRPO implementation, an 83% reduction, while reaching 27.2% instead of 27.6% accuracy. The underlying insight is that the optimal batch sizes for inference and backpropagation are different, so an efficient RL loop should decouple them.

What carries the argument

The mechanism is the pair of efficiency operations inside DASH. Preemptive sampling decouples the inference batch size (256) from the backpropagation batch size (realized through gradient accumulation in 32 steps), so each phase runs at its own hardware-optimal scale; the observation that inference is memory-light while backpropagation is memory-heavy justifies this split. Gradient filtering drops samples with small advantage estimates (threshold 0.1), where the advantage uses GRPO's group-relative baseline $A(\hat y_n \mid x_n) \approx r_n - \frac{1}{|N_k|}\sum_{n' \in N_k} r_{n'}$, the reward of one generation minus the mean reward of other generations of the same prompt. Together, the two operations make the policy-gradient update much cheaper per sampled token without changing which samples dominate the learning signal.

What would settle it

Run the same GRPO implementation with DASH's large inference batch (256) while keeping the training batch at 8; if wall-clock time falls toward 6-7 hours, the 83% speedup is a batch-size artifact, not a property of preemptive sampling plus gradient filtering. Run both algorithms over several seeds on MATH-500 and check whether 27.6% and 27.2% are within one standard error.

Watch

Extended reading notes

Core claim

The central discovery is that the sampling step, not the gradient step, dominates wall-clock time in on-policy RL for reasoning, because inference uses memory far more efficiently than backpropagation and therefore wants a much larger batch. DASH responds by sampling a large batch on inference workers, caching the generations, and then applying many small gradient-accumulation steps on separate backpropagation workers; it then filters out samples whose advantage estimate is below 0.1 so backpropagation never wastes time on them. With GRPO's group-relative advantage baseline and a single policy-gradient step per sampled batch, DASH matches GRPO's accuracy on the measured settings while cutting training time from 38.9 to 6.6 hours on Qwen2.5-0.5B on MATH. The same experiments indicate that on-policy RL outperforms SFT on these small base models, that PPO-style mini-batch updates can reach slightly higher accuracy but with highly unstable reward curves, and that dropping the KL term improves accuracy in most cells and shortens average generation length.

Load-bearing premise

The central claim presumes that the GRPO baseline is a fair, standard implementation with its default small inference batch, so the 38.9-hour versus 6.6-hour gap, and the 0.4-point accuracy difference, reflect the algorithm rather than an avoidable configuration choice.

Editorial extensions

If this is right

  • If the 83% figure holds, on-policy RL for reasoning becomes roughly six times cheaper, which changes the compute budget needed to train small reasoning models.
  • The decoupling idea is not specific to GRPO: any on-policy algorithm that samples large batches can adopt preemptive sampling, and any algorithm with a thresholdable advantage can adopt filtering.
  • Because gradient filtering removes samples with near-zero advantage, its savings should grow as training saturates and more generations become either consistently correct or consistently wrong.
  • No-KL runs were both more accurate and shorter-generation, so removing KL regularization compounds DASH's efficiency gain rather than trading accuracy for speed.
  • The smaller coding speedup (35.3 to 16.3 minutes on MBPP+) indicates the method's benefit scales with generation length, so longer-horizon reasoning tasks should see the largest gains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the reported accuracy parity (27.2% vs 27.6%) rests on a single run; repeated-seed experiments would be needed to confirm that the two methods differ only in speed, not in quality.
  • Editorial inference: the paper's own appendix shows gradient filtering saves 10% at per-device batch 4 versus 4% at batch 2, so the filtering benefit appears to grow with batch size; tuning the threshold may yield larger savings on bigger hardware.
  • Editorial inference: removing KL divergence also concentrates probability mass, and the paper notes reduced diversity at larger k; users who care about diverse reasoning traces may want to keep some KL penalty even though it costs accuracy.
  • Editorial inference: the batch-size mismatch insight suggests that other compute-heavy post-training loops, such as RL with process rewards or search-based self-imitation, could inherit the same preemptive-sampling speedup.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper empirically studies RL design choices for improving mathematical and code reasoning in small (0.5B-3B) Qwen2.5 base models on MATH-500, GSM8K, and MBPP+. It reports four findings: on-policy RL clearly beats SFT (from human or 7B-instruct traces) on in-distribution accuracy; PPO-style multi-step/mini-batch updates give small accuracy gains at 0.5B but visibly unstable training curves; removing KL regularization shortens generations and improves in-distribution accuracy; and a proposed method, DASH (preemptive sampling with a large inference batch plus gradient filtering that drops low-advantage samples), reduces wall-clock training time by 83% relative to a Huggingface-GRPO baseline (38.9h to 6.6h) at apparently equal MATH accuracy. All results are single-run point estimates; code is released.

Significance. Credit: the paper is well-organized, open-sources its implementation, includes useful ablations (No-GF, No-KL, pass@k for KL), and its practical diagnosis that inference and backpropagation prefer different batch sizes is plausible and potentially widely applicable. If the 83% figure survives a controlled comparison, the engineering recipe is valuable for compute-limited small-model RL. However, the headline speedup is driven almost entirely by the batching change (No-GF alone is 6.9h), the GRPO baseline is not given the large inference batch that Section 3.4 itself identifies as the bottleneck, and the accuracy-parity claim has no error bars and is contradicted by the authors' own Appendix B numbers. The PPO finding is likewise stated more strongly than Table 5 supports. The findings are worth publishing after the controlled comparison and claim-qualification are added.

major comments (4)
  1. [Section 4.3, Table 3, Appendix A] The headline 83% speedup is confounded with the inference batch size. The GRPO baseline samples with the backpropagation batch of 8, while DASH uses an inference batch of 256 (128 per inference GPU), and no GRPO control with a large inference batch is reported. Preemptive sampling alone (No-GF, 6.9h) accounts for (38.9-6.9)/38.9, roughly 82%, of the reduction; gradient filtering adds only about 4%. Because Section 3.4 argues that the inference/backprop batch-size mismatch is the bottleneck, a fair baseline must be allowed the same inference batching; note that Multi already uses M=256, so its wall-clock time would provide such a control, but that time is not reported. As it stands, the 83% measures a configuration difference, not a property of the DASH algorithm.
  2. [Section 4.3, Table 3, Appendix B, Table 10] The accuracy-parity claim rests on single runs with no seeds or confidence intervals. DASH versus GRPO shows MATH 27.2 versus 27.6 (0.4 points) and GSM8K 31.1 versus 32.8 (1.7 points), and the GSM8K gap is large relative to the DASH improvement over base (31.1 versus 30.3). More importantly, Appendix B, Table 10 reports DASH at 28.4 MATH versus No-GF at 31.8 with per-device batch 4, a 3.4-point drop that the text does not acknowledge; the text says only that gradient filtering is 'more effective' at batch 4 (in wall-clock time). A claim of no accuracy loss requires error bars, and the paper's own appendix numbers are inconsistent with the unqualified 'without sacrificing accuracy' statement.
  3. [Section 4.4, Table 5, Abstract] The claim that 'PPO-based off-policy updates increase accuracy instead of reduce variance' is contradicted by the Mini 1.5B row: MATH 36.0 versus DASH 54.0 and GSM8K 8.1 versus 58.8, both below the base model (48.0 and 58.8). Variance is never measured, since there are no repeated runs anywhere in the paper; 'significantly higher variance' is inferred only from qualitative training curves in Figure 4. In addition, the 1.5B Multi and Mini runs use a learning rate of 3e-06 while DASH uses 1e-06 (Table 7), confounding the update rule with the learning rate. The abstract and the introduction's finding (ii) need to be qualified to reflect the 0.5B results and the acknowledged Mini collapse at 1.5B.
  4. [Section 3.5, Table 7, Figure 3, Appendix B] The gradient filtering mechanism is not characterized as implemented. With binary rewards and 4 generations per prompt, the group-mean advantage in Equation (4) takes values in {0, +/-0.25, +/-0.5, +/-0.75}, so the threshold 0.1 drops exactly the zero-advantage samples and leaves all nonzero advantages untouched. Consequently, the kept gradient is identical to the No-GF gradient, the increase in mean absolute advantage in Figure 3a is mechanical, and the general framing about dropping 'small advantage estimates' is not exercised. No sensitivity analysis for the threshold is given, and the asserted heuristic that large policy-gradient norms are unlikely is unsupported. This also means the 3.4-point MATH gap in Table 10 cannot be explained by the filter itself, so it must reflect run-to-run noise or an unmodeled effect, which underscores the need for repeated runs.
minor comments (5)
  1. [Section 4.2, Table 1, Section 4.5, Table 6] The statement that 'DASH improves performance both in-distribution and out-of-distribution' is contradicted by the 3B GSM8K row in Table 1 (64.6 versus base 66.0), and the abstract's 'removing KL divergence ... higher accuracy' is contradicted by Table 6, where No-KL 3B GSM8K is 60.0 versus DASH's 64.6. Both claims should be qualified as in-distribution or as dataset-dependent.
  2. [Figure 1, Abstract, Table 4] The '83%' figure applies only to the 0.5B MATH setting; the coding experiments in Table 4 show a speedup of roughly 53% (35.3 to 16.5 minutes). The scope of the headline number should be stated in the abstract and figure caption.
  3. [Section 3.2, Table 7] The KL coefficient beta in the definition J_DASH = J_PG + beta*J_KL is never given a value, and Table 7 does not list it. Since the KL term is included in DASH and removed in No-KL, the actual beta used should be reported (the text only says other parameters use Huggingface trainer defaults).
  4. [Appendix A, Table 7] The GPU allocation row for Qwen2.5-0.5B reads '4 / 2 (2 / 2 using ZeRO)' and is unclear, and the GRPO baseline's inference batch size should be stated explicitly as equal to the 8-sample backpropagation batch rather than left implicit, so the asymmetry with DASH's 256 is transparent.
  5. [Throughout] There are several typos: 'demonstrating the the general effectiveness' (Section 4.2, coding paragraph), 'hyperparamaters' (Appendix A), and the Figure 3 caption 'Training reward recurves' should read 'curves'. Also, the trl version is described only by a clone date (April 10, 2025); pinning the commit would improve reproducibility.

Circularity Check

1 steps flagged · score 1.0 of 10

Central claims are empirical comparisons against external benchmarks; only a minor, non-load-bearing self-definitional observation in Figure 3a.

  1. self definitional [Section 3.5 (gradient filtering definition); Section 4.3 / Figure 3a caption and surrounding text]
    "we propose to drop examples with small advantage estimates (which is equivalent to clipping small advantage values to zero, effectively dropping them from the gradient update). ... As shown in Figure 3a, gradient filtering increases the average absolute advantage values, leading to more significant gradient updates."

    Gradient filtering is defined (Section 3.5) as dropping samples with small-magnitude advantages, so the mean absolute advantage of the surviving samples cannot decrease; the increase highlighted in Figure 3a is a logical consequence of the filter's definition rather than an independent empirical discovery. The paper uses it to explain more significant gradient updates, but the actual claimed benefits (6.6h vs 6.9h wall-clock in Table 3; 27.2 vs 27.4 MATH) are separately measured, so this tautology is not load-bearing for the central claims.

full rationale

The paper's headline results are empirical comparisons against external benchmarks or internally measured wall-clock times: on-policy RL vs SFT (Tables 1-2, MATH-500/GSM8K/MBPP+), DASH's 83% runtime reduction (Table 3: 38.9h vs 6.6h), accuracy parity (27.6 vs 27.2 MATH, 32.8 vs 31.1 GSM8K), the PG-vs-PPO instability comparison (Table 5), and the KL-removal findings (Table 6). No equation is fitted to the reported outcome, no parameter is tuned on the evaluation splits, and no uniqueness theorem or prior-work derivation is invoked to force DASH's design. The only self-citation (Shypula et al., 2025, for the intuition that RL concentrates probability mass) is non-load-bearing and appears alongside an external citation (West and Potts, 2025). The single mildly self-definitional passage is Figure 3a's claim that gradient filtering increases the average absolute advantage; because the filter is defined as dropping small-advantage samples, that increase follows by construction. It functions as an explanatory illustration, and the actual 4% speedup of DASH over No-GF and the accuracy numbers are measured. The Limitations section confirms that hyperparameters reflect commonly used values rather than values fitted to reported outcomes. The skeptic's concerns, regarding the GRPO baseline's small effective inference batch (Appendix A: training batch 8 with no separate large inference batch, versus DASH's inference batch 256) and single-run comparisons without error bars, are baseline-fairness and statistical-validity questions, not circularity. The derivation chain is otherwise self-contained, so the circularity score is 1.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central efficiency claim rests on a small number of hand-chosen constants (batch size, filtering threshold, KL coefficient) and on two empirical regularities: small-advantage samples are low-information, and inference/backpropagation prefer different batch sizes. No new physical or conceptual entities are introduced.

free parameters (3)
  • Gradient filtering threshold = 0.1
    Fixed by hand with no sensitivity sweep; Appendix B shows its effect changes with batch size (4% at batch 2, 10% at batch 4), so the headline time decomposition is sensitive to this chosen constant.
  • DASH inference batch size = 256 (128 per inference GPU; 1024 for the 1.5B PPO comparison)
    Chosen for hardware rather than derived; it is the core of preemptive sampling and directly determines the wall-clock speedup.
  • KL regularization coefficient beta = unspecified, HuggingFace Trainer default
    DASH includes a KL term for comparison with PPO variants, but the value is not reported; the No-KL comparison depends on this unstated constant.
assumptions (4)
  • standard math Policy Gradient Theorem gives an unbiased gradient estimate (Eq. 2).
    Used in Section 3.2 as the basis for the DASH update; this is standard background math.
  • domain assumption Reward is a scalar, typically a binary correctness indicator for the whole trajectory.
    Stated in Section 3 and footnote 2; process rewards are dismissed based on prior work, and all results depend on this simplification.
  • ad hoc to paper Samples with small advantage estimates contribute negligible gradient because large policy-gradient norms are unlikely.
    Section 3.5 asserts this is 'unlikely in practice' without measurement; it is load-bearing for the correctness of gradient filtering.
  • domain assumption Optimal batch sizes for inference and backpropagation differ.
    Section 3.4 relies on this to justify preemptive sampling; it is supported by memory characteristics but not formalized or measured in isolation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Effective Reinforcement Learning for Reasoning in Language Models." pith.science (2026). https://pith.science/paper/3PD4QSV2

@misc{pith2026250517218,
  author       = {Pith},
  title        = {Pith review of: Effective Reinforcement Learning for Reasoning in Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3PD4QSV2}},
  note         = {Machine review of arXiv:2505.17218}
}
read the original abstract

Reinforcement learning (RL) has emerged as a promising strategy for improving the reasoning capabilities of language models (LMs) in domains such as mathematics and coding. However, most modern RL algorithms were designed to target robotics applications, which differ significantly from LM reasoning. We analyze RL algorithm design decisions for LM reasoning, for both accuracy and computational efficiency, focusing on relatively small models due to computational constraints. Our findings are: (i) on-policy RL significantly outperforms supervised fine-tuning (SFT), (ii) PPO-based off-policy updates increase accuracy instead of reduce variance, and (iii) removing KL divergence can lead to more concise generations and higher accuracy. Furthermore, we find that a key bottleneck to computational efficiency is that the optimal batch sizes for inference and backpropagation are different. We propose a novel algorithm, DASH, that performs preemptive sampling (i.e., sample a large batch and accumulate gradient updates in small increments), and gradient filtering (i.e., drop samples with small advantage estimates). We show that DASH reduces training time by 83% compared to a standard implementation of GRPO without sacrificing accuracy. Our findings provide valuable insights on designing effective RL algorithms for LM reasoning.

Figures

Figures reproduced from arXiv: 2505.17218 by the authors.

Figure 1
Figure 1. DASH can reduce running time by 83% com [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Illustration of preemptive sampling. We use [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. Training reward curves for PG vs. PPO on [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figures from the paper (6 more)
Figure 3
Figure 3. Figure 3: Comparison between DASH and No-GF for Qwen2.5-0.5B on math. 4.5 KL Divergence Regularization Next, we compare DASH to an ablation without the KL divergence term, denoted No-KL. Training reward curves are shown in Figure 6a. As can be seen, removing KL divergence regula…
Figure 5
Figure 5. Figure 5: Impact of KL divergence regularization on [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Comparing KL divergence regularization on [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: DeepSpeed configuration. Versions of python and key libraries are shown in [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Training reward curves for PG vs. PPO on [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 9
Figure 9. Figure 9: Comparison of No-GF and DASH with a per [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SLPO: Scaling Latent Reasoning via a Surrogate Policy

    cs.CL 2026-07 conditional novelty 6.0 of 10

    SLPO adds a surrogate Gaussian policy and a learnable stopping gate so that outcome-reward RL can improve latent (continuous-vector) reasoning, raising Pass@8/16 in all 12 tested settings.

Reference graph

Works this paper leans on

41 extracted references · 3 canonical work pages · cited by 1 Pith paper

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. 2024. https://arxiv.org/abs/2402.14740 Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms . Preprint, arXiv:2402.14740

    arXiv 2024

  4. [4]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021. https://arxiv.org/abs/2108.07732 Program synthesis with large language models . Preprint, arXiv:2108.07732

    arXiv 2021

  5. [5]

    Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, and Torsten Hoefler. 2024. https://doi.org/10.1609/aaai.v38i16.29720 Graph of thoughts: Solving elaborate problems with large language models . Proceedings of the AAAI Conference on Artificial ...

    doi:10.1609/aaai.v38i16.29720 2024

  6. [6]

    Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao. 2023. https://arxiv.org/abs/2310.05915 Fireact: Toward language agent fine-tuning . Preprint, arXiv:2310.05915

    arXiv 2023

  7. [7]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, and 39 others. 2021. https://arxiv.org/abs/2107.03374 Evaluating large lang...

    arXiv 2021

  8. [8]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. https://arxiv.org/abs/2110.14168 Training verifiers to solve math word problems . Preprint, arXiv:2110.14168

    arXiv 2021

Show all 41 references
  1. [9]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, and 181 others. 2025 a . https://arxiv.org/abs/2501....

    2025 arXiv

  2. [10]

    DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, and 181 others. 2025 b . https://arxiv.org/abs...

    2025 arXiv

  3. [11]

    Elliot Glazer, Ege Erdil, Tamay Besiroglu, Diego Chicharro, Evan Chen, Alex Gunning, Caroline Falkman Olsson, Jean-Stanislas Denain, Anson Ho, Emily de Oliveira Santos, Olli Järviniemi, Matthew Barnett, Robert Sandler, Matej Vrzala, Jaime Sevilla, Qiuyu Ren, Elizabeth Pratt, L...

    2024 arXiv

  4. [12]

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. https://arxiv.org/abs/2103.03874 Measuring mathematical problem solving with the math dataset . Preprint, arXiv:2103.03874

    2021 arXiv

  5. [13]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. https://arxiv.org/abs/2106.09685 Lora: Low-rank adaptation of large language models . Preprint, arXiv:2106.09685

    2021 arXiv

  6. [14]

    Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. 2019. https://arxiv.org/abs/1907.00456 Way off-policy batch deep reinforcement learning of implicit human preferences in dialog . Preprint, arXi...

    2019 arXiv

  7. [15]

    Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, and Nicolas Le Roux. 2024. https://arxiv.org/abs/2410.01679 Vineppo: Unlocking rl potential for llm reasoning through refined credit assignment . Preprint, arXiv:2410.01679

    2024 arXiv

  8. [16]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023 a . https://arxiv.org/abs/2309.06180 Efficient memory management for large language model serving with pagedattention . Preprint, arXiv:2309.06180

    2023 arXiv

  9. [17]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023 b . Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operatin...

    2023

  10. [18]

    Litian Liang, Yaosheng Xu, Stephen McAleer, Dailin Hu, Alexander Ihler, Pieter Abbeel, and Roy Fox. 2022. https://arxiv.org/abs/2209.07670 Reducing variance in temporal-difference value estimation via ensemble of deep networks . Preprint, arXiv:2209.07670

    2022 arXiv

  11. [19]

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. https://arxiv.org/abs/2305.20050 Let's verify step by step . Preprint, arXiv:2305.20050

    2023 arXiv

  12. [20]

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. https://openreview.net/forum?id=1qvx610Cu7 Is your code generated by chat GPT really correct? rigorous evaluation of large language models for code generation . In Thirty-seventh Conference on Neural Informa...

    2023

  13. [21]

    Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. 2025. https://arxiv.org/abs/2503.20783 Understanding r1-zero-like training: A critical perspective . Preprint, arXiv:2503.20783

    2025 arXiv

  14. [22]

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. 2025. https://arxiv.org/abs/2501.19393 s1: Simple test-time scaling . Preprint, arXiv:2501.19393

    2025 arXiv

  15. [23]

    Junhyuk Oh, Yijie Guo, Satinder Singh, and Honglak Lee. 2018. https://arxiv.org/abs/1806.05635 Self-imitation learning . Preprint, arXiv:1806.05635

    2018 arXiv

  16. [24]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing sys...

    2022

  17. [25]

    Ruchir Puri, David S. Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladimir Zolotov, Julian Dolby, Jie Chen, Mihir Choudhury, Lindsey Decker, Veronika Thost, Luca Buratti, Saurabh Pujar, Shyam Ramji, Ulrich Finkler, Susan Malaika, and Frederick Reiss. 2021. https://arxi...

    2021 arXiv

  18. [26]

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, and 25 others. 2025. https://arxiv.org/abs/2412.15115 Qwe...

    2025 arXiv

  19. [27]

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. https://arxiv.org/abs/1910.02054 Zero: Memory optimizations toward training trillion parameter models . Preprint, arXiv:1910.02054

    2020 arXiv

  20. [28]

    Jordan, and Pieter Abbeel

    John Schulman, Sergey Levine, Philipp Moritz, Michael I. Jordan, and Pieter Abbeel. 2017 a . https://arxiv.org/abs/1502.05477 Trust region policy optimization . Preprint, arXiv:1502.05477

    2017 arXiv

  21. [29]

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017 b . https://arxiv.org/abs/1707.06347 Proximal policy optimization algorithms . Preprint, arXiv:1707.06347

    2017 arXiv

  22. [30]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. https://arxiv.org/abs/2402.03300 Deepseekmath: Pushing the limits of mathematical reasoning in open language models . Preprint, arXiv:2402.03300

    2024 arXiv

  23. [31]

    Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. https://arxiv.org/abs/2303.11366 Reflexion: Language agents with verbal reinforcement learning . Preprint, arXiv:2303.11366

    2023 arXiv

  24. [32]

    Alexander Shypula, Shuo Li, Botong Zhang, Vishakh Padmakumar, Kayo Yin, and Osbert Bastani. 2025. https://arxiv.org/abs/2504.12522 Evaluating the diversity and quality of llm generated content . Preprint, arXiv:2504.12522

    2025

  25. [33]

    Sutton and Andrew G

    Richard S. Sutton and Andrew G. Barto. 2018. Reinforcement Learning: An Introduction. A Bradford Book, Cambridge, MA, USA

    2018

  26. [34]

    Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. 1999. https://proceedings.neurips.cc/paper_files/paper/1999/file/464d828b85b0bed98e80ade0a5c43b0f-Paper.pdf Policy gradient methods for reinforcement learning with function approximation . In Advances in N...

    1999

  27. [35]

    Peiyi Wang, Lei Li, Zhihong Shao, R. X. Xu, Damai Dai, Yifei Li, Deli Chen, Y. Wu, and Zhifang Sui. 2024. https://arxiv.org/abs/2312.08935 Math-shepherd: Verify and reinforce llms step-by-step without human annotations . Preprint, arXiv:2312.08935

    2024 arXiv

  28. [36]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. https://arxiv.org/abs/2201.11903 Chain-of-thought prompting elicits reasoning in large language models . Preprint, arXiv:2201.11903

    2023 arXiv

  29. [37]

    Peter West and Christopher Potts. 2025. https://arxiv.org/abs/2505.00047 Base models beat aligned models at randomness and creativity . Preprint, arXiv:2505.00047

    2025 arXiv

  30. [38]

    Griffiths, Yuan Cao, and Karthik Narasimhan

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. 2023 a . https://arxiv.org/abs/2305.10601 Tree of thoughts: Deliberate problem solving with large language models . Preprint, arXiv:2305.10601

    2023 arXiv

  31. [39]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023 b . https://arxiv.org/abs/2210.03629 React: Synergizing reasoning and acting in language models . Preprint, arXiv:2210.03629

    2023 arXiv

  32. [40]

    Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. 2023. https://arxiv.org/abs/2310.12823 Agenttuning: Enabling generalized agent abilities for llms . Preprint, arXiv:2310.12823

    2023 arXiv

  33. [41]

    Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. 2025. https://arxiv.org/abs/2503.18892 Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild . Preprint, arXiv:2503.18892

    2025 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.