Pith. sign in

REVIEW 4 major objections 6 minor 2 cited by

SuperRL: Reinforcement Learning with Supervision to Boost Language Model Reasoning

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SuperRL claims that per-instance reward gating—apply policy gradients when any rollout succeeds, supervised imitation when all rollouts fail—makes language-model reasoning training more sample-efficient, generalizable, and robust under…

desk verdict A genuinely simple and new reward-gated SFT fallback, but the experiments never isolate adaptivity from simply adding more SFT data—worth refereeing, not yet convincing. read the letter →

arxiv 2506.01096 v2 pith:6P23HH4C submitted 2025-06-01 cs.AI

Yihao Liu , Shuocheng Li , Lang Cao , Yuhang Xie , Mengyu Zhou , Haoyu Dong , Xiaojun Ma , Shi Han
show 1 more author
Dongmei Zhang
This is my paper
classification cs.AI
keywords largelanguagemodelsreinforcementlearningsupervisedfine-tuningrewardgatingsparserewardsreasoningbenchmarksGRPOsampleefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that online reinforcement learning for LLM reasoning becomes substantially more reliable when the reward signal itself decides, per question, whether to update with policy gradients or with supervised fine-tuning. The proposed rule is simple: if at least one sampled answer gets a nonzero reward, apply the usual RL update; if every sampled answer gets zero reward, imitate a curated expert solution instead. The authors argue this makes sparse-reward training more sample-efficient, improves out-of-distribution generalization, and stabilizes training, without introducing new hyperparameters. Evidence comes from cross-dataset experiments on math and table reasoning benchmarks, together with the caveat that the gate fails on two data regimes where a single success is not a trustworthy signal.

What carries the argument

The load-bearing object is the reward-based soft gate $c(x)=\mathbf{1}[\max_k R(x,y_k)>0]$, a binary, hyperparameter-free switch that selects per instance between the supervised loss $L_{\mathrm{SFT}}(\theta;x)=-(1/M)\sum_m \log \pi_\theta(y^{*(m)}|x)$ and any monotone policy-gradient surrogate $L_{\mathrm{PG}}(\theta;x)$, such as PPO, TRPO, or GRPO (Group Relative Policy Optimization). The unified objective is $L_{\mathrm{SuperRL}}(\theta;x)=(1-c(x))L_{\mathrm{SFT}}(\theta;x)+c(x)L_{\mathrm{PG}}(\theta;x)$. The gate's work is to convert the presence of at least one successful rollout into a decision that RL is informative and the absence of any success into a decision that imitation is safer; the paper also ablates an advantage-based variant $c_A(x)=\mathbf{1}[\max_k A_k>0]$ and an uncertainty-weighted soft fusion $L_{\mathrm{Hybrid}}=e^{-2\sigma_{\mathrm{pg}}}L_{\mathrm{PG}}+e^{-2\sigma_{\mathrm{sft}}}L_{\mathrm{SFT}}+\sigma_{\mathrm{pg}}+\sigma_{\mathrm{sft}}$.

What would settle it

A controlled experiment that changes the gate to require at least two successful rollouts before switching off supervised fine-tuning, run on HiTab or LIMO: if the stricter gate beats the one-success gate on test accuracy, then the claim that a single nonzero reward is a sufficient signal for reinforcement learning is false.

Watch

Extended reading notes

Core claim

The central discovery is that instance-level reward gating can unify supervised fine-tuning and reinforcement learning in one update rule. Define $c(x)=\mathbf{1}[\max_k R(x,y_k)>0]$, the indicator that at least one of the $K$ sampled rollouts for input $x$ receives nonzero reward; the training loss is $(1-c(x))L_{\mathrm{SFT}}(x)+c(x)L_{\mathrm{PG}}(x)$. Instances with at least one success get a policy-gradient update; instances where every rollout fails imitate curated offline demonstrations. The paper reports that across $5\times 7$ cross-dataset training/evaluation pairs, SuperRL improves on vanilla RL in 24 cases, ties once, and is worse in 10, with the largest gains (up to +29.9 percentage points) when the source dataset has dense or moderately sparse rewards. It also documents the gate's failure modes: on LIMO (817 hard, heavily filtered examples) the reward gate is too permissive and SuperRL underperforms vanilla RL in-domain, and on HiTab a single successful rollout switches off SFT too early, so two-stage SFT+RL remains the better baseline.

Load-bearing premise

The load-bearing premise is that one successful sampled answer is a dependable sign that a reinforcement-learning update will help that instance more than another round of supervised imitation; the paper's own HiTab and LIMO experiments show this assumption can fail.

Editorial extensions

If this is right

  • On sparse-reward tasks, the SFT fallback activates frequently early in training and phases out as the policy improves, and the paper links this to reduced entropy variance and more stable optimization than vanilla RL.
  • On dense-reward tasks the gate rarely triggers SFT, so SuperRL performs comparably to or better than vanilla RL without sacrificing performance.
  • Across the $5\times 7$ cross-dataset grid, SuperRL improves over vanilla RL in 24 of 35 training/evaluation pairs, with gains up to +29.9 percentage points on held-out tasks.
  • The two documented failure regimes, LIMO and HiTab, imply that reward-gated switching should be combined with data-aware scheduling, and the paper lists learnable gating as future work.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the gate works as claimed, sparse reward itself becomes an automatic curriculum: the model imitates exactly when it cannot self-improve, which suggests extending the rule to decide not just whether but how much supervision to inject.
  • The HiTab failure points to a testable strengthening: require more than one successful rollout before switching off SFT, or gate on advantage rather than raw reward, as the paper's own advantage-based variant begins to do.
  • The framework's practical value tracks the quality of curated offline data, so bootstrapping fallback demonstrations from self-generated or retrieved traces would test whether the gate can stand without expert data.
  • The same 'all rollouts failed' signal could route agentic or tool-use tasks to imitation of a plan or subgoal demonstration when rewards are episodic and sparse.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes SuperRL, a training framework for LLM reasoning that switches at the instance level between policy-gradient updates (PPO/GRPO) and supervised fine-tuning on offline demonstrations. For each input, the model samples K rollouts; if any rollout receives a nonzero reward, a policy-gradient loss is applied, and if all rollouts receive zero reward, an SFT loss on curated offline traces is applied. The method is evaluated on five source datasets (GSM8K, Metamath, LIMO, PRM12K, OpenR1) and transferred to seven target benchmarks, with additional experiments on HiTab and on two variants (Advantage-gated and Log-Sigma hybrid). The paper claims improved sample efficiency, generalization, and robustness relative to vanilla RL, and reports 24 improvements, 10 degradations, and 1 tie across 35 source-target transfer pairs, while acknowledging failures on LIMO and HiTab.

Significance. If the central claim holds, SuperRL is an attractively simple, hyperparameter-free mechanism for injecting offline supervision into online RL, and the paper's empirical scope across multiple model families and benchmarks is useful. The paper deserves credit for honestly documenting negative results in Appendix D (offline rollouts as GRPO-compatible data, few-shot prompting), which increases confidence in the authors' reporting. However, the current evidence does not isolate the effect of adaptivity, does not quantify run-to-run variability, and does not support the broad 'sample efficiency' claim, because SuperRL consumes additional offline data and SFT gradient updates that vanilla RL never sees. The paper's own negative results on LIMO and HiTab further qualify the abstract's global claim. The method is plausible and the direction is interesting, but the experimental attribution is not yet established.

major comments (4)
  1. [§3.2, §4.1] The paper's central claim is that the adaptive reward gate—not merely the addition of SFT signal—drives the observed gains. The loss in Eq. (3.2) is L = (1−c(x))L_SFT + c(x)L_PG, and the text in §4.1 states that 'Fixed-schedule hybrids with manually tuned schedules' were ablated and 'yield some improvements over naïve RL,' but no results for this baseline appear in Tables 1–5. The only comparisons are to pure RL, pure SFT, sequential SFT+RL, Adv-Gated, and Log-Sigma. Sequential SFT+RL concentrates all SFT updates before RL and does not match SuperRL's interleaving or total SFT exposure. Consequently, the reported gains could be attributed to the extra offline supervision and additional gradient updates rather than to the reward-gated adaptivity. A matched control that applies the same number of SFT updates on a fixed schedule or on randomly selected instances is required to support the attribution claim.
  2. [§C.7, Tables 1–2] No error bars, confidence intervals, or multiple-seed results are reported for any of the main tables. In §C.7 the final-score selection rule is also post hoc: 'we report the test-set performance at step 500 if the learning curve shows smooth convergence' and otherwise apply EMA smoothing. This procedure can systematically select the most favorable evaluation point and, combined with single runs, makes the 24/10/1 tally in §4.2 difficult to interpret. The robustness claim in the abstract and conclusion requires variance estimates and a pre-specified selection rule.
  3. [§4.1, §C.7] The 'sample efficiency' claim is confounded. SuperRL uses offline demonstrations for SFT fallback and performs additional SFT gradient updates on those demonstrations, whereas the vanilla RL baseline only sees online rollouts. The paper does not report wall-clock time, total tokens consumed, or number of gradient updates matched across methods. Thus 'higher sample efficiency' may simply reflect that SuperRL has access to more data and more updates. A matched-budget comparison, or a clear statement of what 'sample efficiency' means in this setting, is needed.
  4. [§4.2, Tables 1–2] The paper's abstract and introduction claim that SuperRL 'surpasses vanilla RL' and delivers 'improved robustness' broadly, but the paper's own results show substantial negative cases. When trained on LIMO, SuperRL is worse than RL on 4 of 7 transfer targets (Table 1, LIMO row), and on HiTab it underperforms both RL (18.47 vs 30.03 in-domain) and the SFT+RL baseline (38.64) (Table 2). The discussion in §4.2 attributes this to the gate being too permissive or too aggressive, but this means the method's cornerstone assumption—that a single nonzero reward indicates that PG is more beneficial than SFT—is violated in two of the six source datasets. The broad claim in the abstract should be qualified accordingly, or the scope of the claim should be narrowed.
minor comments (6)
  1. [§4.3, Figure 2] The caption for Figure 2 states 'segments with SFT triggered per 500 steps,' but the right panel labels the axis 'Activation Density (segments/600 steps)' and the text refers to 'segments/600 steps'; these should be made consistent.
  2. [References, Table 6] PRM12K is cited as [31], but reference [31] is the Plan-and-Solve paper (Wang et al., 2023), which is not the source of PRM12K; this citation appears to be wrong and should be corrected.
  3. [§3.2–§3.3] Equation numbering is duplicated: Section 3.2 defines Eqs. (1) and (2), and Section 3.3 again labels the Adv-Gated loss and the Log-Sigma loss as (1) and (2). Please renumber the equations consistently.
  4. [Appendix E] Appendix E contains large amounts of corrupted text (e.g., '/uni00000013 /uni00000015/uni00000013/...') that makes the training graph analysis unreadable; these fragments should be removed or replaced with the intended figure references and text.
  5. [Abstract, §5] The abstract and conclusion state that SuperRL 'surpasses vanilla RL' and achieves 'superior performance' without noting that it underperforms on LIMO and HiTab in several settings; a more precise summary that acknowledges these exceptions would improve accuracy.
  6. [§4.3, Table 4] The reported correlation (r = −0.86, p = 0.028) is computed over only six datasets; this should be reported with the small sample size made explicit, and the p-value should be interpreted with caution given the number of correlations examined.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reward-gated switch is an externally evaluated training rule, not a fitted prediction.

full rationale

SuperRL's central objective is L_SuperRL(θ;x) = (1−c(x))L_SFT(θ;x) + c(x)L_PG(θ;x) with c(x)=1[max_k R(x,y_k)>0]. The gate is computed from rollout rewards, and the same rewards feed the policy-gradient term; this is a definition of the proposed training algorithm, not a derivation of the reported results. The paper's claims are empirical: held-out accuracy on GSM8K/MetaMath/PRM12K/LIMO/OpenR1/HiTab, cross-dataset transfer, entropy statistics, and SFT-trigger counts are all measured after training and are not encoded in c(x) or in the loss formula. No parameter is fitted to the target benchmark and then reported as a prediction; the only learnable quantities (σ_pg, σ_sft) appear in the optional Log-Sigma ablation, which is explicitly a variant and not the headline method. Self-citations are not load-bearing: the HiTab reference [33] shares authors but is used solely as a benchmark/dataset source, and the paper reports that SuperRL underperforms on it, so the citation cannot be doing justificatory work. The absence of a fixed-schedule SFT+RL control matched for total SFT updates is a real experimental-attribution limitation, but it is a control/confound issue, not circularity: the gains are not equivalent to the inputs by construction. No reduction of a claimed result to its own inputs, and no self-citation chain forcing the conclusion, was found.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central hard-gate method introduces no new entities and no fitted parameters for the switch itself. The two sigma parameters appear only in the optional Log-Sigma hybrid variant that is compared as an ablation. The main assumptions concern the reliability of the reward signal and the quality of the offline demonstrations.

free parameters (2)
  • sigma_pg (Log-Sigma variant) = not reported
    Learnable log-variance scalar weighting the policy-gradient loss in the Hybrid-Log-Sigma ablation (Section 3.3, Eq. 2). Not part of the central hard-gate SuperRL method.
  • sigma_sft (Log-Sigma variant) = not reported
    Learnable log-variance scalar weighting the SFT loss in the Hybrid-Log-Sigma ablation (Section 3.3, Eq. 2). Not part of the central hard-gate SuperRL method.
assumptions (3)
  • domain assumption Presence of any nonzero reward indicates that a policy-gradient update is more useful than further supervised imitation for that instance.
    Defines the gate c(x) in Section 3.2 and is the core mechanism of SuperRL; contradicted by the paper's own HiTab and LIMO results.
  • domain assumption Offline demonstrations are high-quality and beneficial to imitate when all rollouts fail.
    The SFT fallback in Section 3.2 presumes the curated offline data provides correct reasoning traces worth imitating.
  • domain assumption The reward function (binary exact match after canonicalization) is a reliable indicator of reasoning quality.
    Used throughout training and evaluation (Appendix C.5, C.6); if rewards are noisy or mis-specified, the gate and the reported gains lose meaning.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SuperRL: Reinforcement Learning with Supervision to Boost Language Model Reasoning." pith.science (2026). https://pith.science/paper/6P23HH4C

@misc{pith2026250601096,
  author       = {Pith},
  title        = {Pith review of: SuperRL: Reinforcement Learning with Supervision to Boost Language Model Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6P23HH4C}},
  note         = {Machine review of arXiv:2506.01096}
}
read the original abstract

Large language models are increasingly used for complex reasoning tasks where high-quality offline data such as expert-annotated solutions and distilled reasoning traces are often available. However, in environments with sparse rewards, reinforcement learning struggles to sample successful trajectories, leading to inefficient learning. At the same time, these offline trajectories that represent correct reasoning paths are not utilized by standard on-policy reinforcement learning methods. We introduce SuperRL, a unified training framework that adaptively alternates between RL and SFT. Whenever every rollout for a given instance receives zero reward, indicating the absence of a learning signal, SuperRL falls back to SFT on the curated offline data. Extensive experiments across diverse reasoning benchmarks show that SuperRL surpasses vanilla RL by delivering higher sample efficiency, stronger generalization, and improved robustness under sparse rewards.

Figures

Figures reproduced from arXiv: 2506.01096 by the authors.

Figure 1
Figure 1. Overview of SuperRL. SuperRL is a unified training framework that adaptively combines RL and SFT based on reward signal. During training, for each input, the model samples multiple rollouts and computes their rewards. If at least one trajectory receives a nonzero reward, standard RL updates are applied using policy gradients. If all trajectories yield zero reward—indicating an absence of learning signal—SuperRL fall… view at source ↗
Figure 2
Figure 2. Analysis of SFT Activation Patterns and Efficiency. Left: The histogram shows the count of supervised fine-tuning (SFT) activations per 100 training steps for two datasets: Metamath and LIMO. Most SFT activations in both cases occur in the early phase of training (0–100 steps), with LIMO exhibiting a denser activation pattern overall. Right: A comparison of SFT activation efficiency metrics across datasets. LIMO sho… view at source ↗
Figure 3
Figure 3. Training curve on the GSM8K dataset with [PITH_FULL_IMAGE:figures/full_fig_p025_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Training curve on the OpenR1 dataset with [PITH_FULL_IMAGE:figures/full_fig_p026_4.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Curriculum Learning for Efficient Chain-of-Thought Distillation via Structure-Aware Masking and GRPO

    cs.LG 2026-02 conditional novelty 6.0 of 10

    A three-stage curriculum (masked reconstruction, GRPO compression, teacher-guided rewriting) distills long chain-of-thought into a concise 3B student, reporting 76.19% on GSM8K (up from 64.90%) with 167 output tokens ...

  2. Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle

    cs.CL 2025-09 conditional novelty 3.0 of 10

    A survey that maps reinforcement learning methods, datasets, benchmarks, and open-source tools across the full training lifecycle of large language models, focusing on verifiable-reward reasoning.

Reference graph

Works this paper leans on

34 extracted references · 10 canonical work pages · cited by 2 Pith papers

  1. [1]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  2. [2]

    Self-consistency improves chain of thought reasoning in language models

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022

    arXiv 2022

  3. [3]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  4. [4]

    Towards understanding chain-of-thought prompting: An empirical study of what matters

    Boshi Wang, Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, and Huan Sun. Towards understanding chain-of-thought prompting: An empirical study of what matters. arXiv preprint arXiv:2212.10001, 2022

    arXiv 2022

  5. [5]

    Automatic chain of thought prompting in large language models

    Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.03493, 2022

    arXiv 2022

  6. [6]

    Graphreason: Enhancing reasoning capabilities of large language models through a graph-based verification approach

    Lang Cao. Graphreason: Enhancing reasoning capabilities of large language models through a graph-based verification approach. arXiv preprint arXiv:2308.09267, 2023

    arXiv 2023

  7. [7]

    A survey on test-time scaling in large language models: What, how, where, and how well?, 2025

    Qiyuan Zhang, Fuyuan Lyu, Zexu Sun, Lei Wang, Weixu Zhang, Wenyue Hua, Haolun Wu, Zhihan Guo, Yufei Wang, Niklas Muennighoff, Irwin King, Xue Liu, and Chen Ma. A survey on test-time scaling in large language models: What, how, where, and how well?, 2025

    2025

  8. [8]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    arXiv 2024

Show all 34 references
  1. [9]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei F...

    2025

  2. [10]

    Proximal policy optimization algorithms, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. 12

    2017

  3. [11]

    Offline reinforcement learning: Tutorial, review, and perspectives on open problems, 2020

    Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems, 2020

    2020

  4. [12]

    Off-policy deep reinforcement learning without exploration, 2019

    Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration, 2019

    2019

  5. [13]

    Hindsight experience replay, 2018

    Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay, 2018

    2018

  6. [14]

    Stanley, and Jeff Clune

    Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. Go-explore: a new approach for hard-exploration problems, 2021

    2021

  7. [15]

    Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M

    Jason Wei, Maarten Bosma, Vincent Y . Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V . Le. Finetuned language models are zero-shot learners, 2022

    2022

  8. [16]

    Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Ni...

    2022

  9. [17]

    Le, Sergey Levine, and Yi Ma

    Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V . Le, Sergey Levine, and Yi Ma. Sft memorizes, rl generalizes: A comparative study of foundation model post-training, 2025

    2025

  10. [18]

    Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B

    Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences, 2020

    2020

  11. [19]

    An empirical study of catastrophic forgetting in large language models during continual fine-tuning, 2025

    Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning, 2025

    2025

  12. [20]

    Understanding catastrophic forgetting in language models via implicit inference, 2024

    Suhas Kotha, Jacob Mitchell Springer, and Aditi Raghunathan. Understanding catastrophic forgetting in language models via implicit inference, 2024

    2024

  13. [21]

    Mitigating forgetting in llm supervised fine-tuning and preference learning, 2025

    Heshan Fernando, Han Shen, Parikshit Ram, Yi Zhou, Horst Samulowitz, Nathalie Baracaldo, and Tianyi Chen. Mitigating forgetting in llm supervised fine-tuning and preference learning, 2025

    2025

  14. [22]

    A minimalist approach to llm reasoning: from rejection sampling to reinforce, 2025

    Wei Xiong, Jiarui Yao, Yuhui Xu, Bo Pang, Lei Wang, Doyen Sahoo, Junnan Li, Nan Jiang, Tong Zhang, Caiming Xiong, and Hanze Dong. A minimalist approach to llm reasoning: from rejection sampling to reinforce, 2025

    2025

  15. [23]

    Reinforcement learning with verifiable rewards: Grpo’s effective loss, dynam- ics, and success amplification, 2025

    Youssef Mroueh. Reinforcement learning with verifiable rewards: Grpo’s effective loss, dynam- ics, and success amplification, 2025

    2025

  16. [24]

    Rlhf workflow: From reward modeling to online rlhf, 2024

    Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf, 2024

    2024

  17. [25]

    Crowder, Darrien M

    Douglas C. Crowder, Darrien M. McKenzie, Matthew L. Trappett, and Frances S. Chance. Hindsight experience replay accelerates proximal policy optimization, 2024

    2024

  18. [26]

    Light-r1: Curriculum sft, dpo and rl for long cot from scratch and beyond, 2025

    Liang Wen, Yunke Cai, Fenrui Xiao, Xin He, Qi An, Zhenyu Duan, Yimin Du, Junchen Liu, Lifu Tang, Xiaowei Lv, Haosheng Zou, Yongchao Deng, Shousheng Jia, and Xiangzheng Zhang. Light-r1: Curriculum sft, dpo and rl for long cot from scratch and beyond, 2025

    2025

  19. [27]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and...

    2022

  20. [28]

    Qwen3 technical report, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, 13 Jianxin Yang, ...

    2025

  21. [29]

    Training verifiers to solve math word problems, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021

    2021

  22. [30]

    Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu

    Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models, 2024

    2024

  23. [31]

    Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models, 2023

    Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models, 2023

    2023

  24. [32]

    Limo: Less is more for reasoning, 2025

    Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning, 2025

    2025

  25. [33]

    Hitab: A hierarchical table dataset for question answering and natural language generation, 2022

    Zhoujun Cheng, Haoyu Dong, Zhiruo Wang, Ran Jia, Jiaqi Guo, Yan Gao, Shi Han, Jian-Guang Lou, and Dongmei Zhang. Hitab: A hierarchical table dataset for question answering and natural language generation, 2022

    2022

  26. [34]

    Let’s think step by step and output the final answer in boxed{}

    The Agentica Team. Deepscaler: Surpassing o1-preview with a 1.5b model by scaling rl, 2024. Technical Report. 14 Contents of Appendix A Limitations and Future Work 16 B Broader Impacts and Safeguards 16 C Detailed Settings of Experiments 17 C.1 Dataset Configurations . . . . ....

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.