Pith. sign in

REVIEW 1 major objections 5 minor 8 cited by

Staying in the Sweet Spot: Responsive Reasoning Evolution via Capability-Adaptive Hint Scaffolding

T0 review · 1 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper tries to establish that RLVR training is fastest when each problem's rollout accuracy is held near 50%, and that adaptive hint lengths can reach that point.

desk verdict The adaptive-hint mechanism is a real, useful empirical contribution; the 50% accuracy rule is plausible but is supported by the ablation, not by the derivation, which only bounds the loss descent by the variance with an unjustified Cramér–Rao step. read the letter →

arxiv 2509.06923 v1 pith:MRBW2T5B submitted 2025-09-08 cs.LG

Ziheng Li , Zexu Sun , Jinman Zhao , Erxue Min , Yongcheng Zeng , Hui Wu , Hengyi Cai , Shuaiqiang Wang
show 3 more authors
Dawei Yin Xu Chen Zhi-Hong Deng
This is my paper
classification cs.LG
keywords reinforcementlearningwithverifiablerewardsreasoninghintscaffoldingitemresponsetheoryrolloutaccuracyGRPOproblemdifficultymathematical
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the learning efficiency of reinforcement learning with verifiable rewards peaks when a training problem is neither too easy nor too hard—specifically, when the model's rollout accuracy on it is around 50%, and that falling away from this sweet spot on either side reduces the guaranteed one-step loss descent. To exploit this, the paper introduces SEELE, which appends a variable-length hint (a prefix of a reference solution) to each problem and, per instance and per training step, chooses the hint length that should bring accuracy to about 50%. The hint length is chosen through several rounds of rollout sampling, fitting a three-parameter logistic item-response model to (hint length, accuracy) pairs, then predicting the next round's length. On six math benchmarks, SEELE improves average accuracy over GRPO by 11.8 points and over SFT by 10.5 points on Qwen2.5 base models, and beats static hint-based baselines. If right, the practical recipe is simple: keep each instance's success probability near coin-flip to make each rollout maximally informative.

What carries the argument

The load-bearing identity is Eq. (5), the quadratic upper bound L(θold)−L(θold+d) ≤ (1/2β) E_x[a(1−a)] connecting guaranteed loss descent to per-instance accuracy; it is derived by Taylor expansion plus the vector Cramér–Rao bound replacing the Fisher-information term with the reward variance. The operating mechanism is a three-parameter logistic (3PL) item-response model f_ϕ(p) = b + (1−b)/(1+e^{−k(p+μ)}) mapping hinting rate p to completion accuracy, fitted by non-linear least squares on multi-round rollout data and inverted to choose the hint length that hits 50% accuracy.

What would settle it

Choose any RLVR checkpoint and bin training instances by their rollout accuracy (say 0–20%, 20–40%, ..., 80–100%). After a single GRPO gradient step from that checkpoint, measure the actual per-instance loss reduction. If the largest descent occurs in a bin away from the 40–60% range, Eq. (5)'s prediction is falsified. A complementary check: compute ∇a^T F^{-1}∇a and Var(r) separately on a small auto-regressive model; if the former exceeds the latter, the Cramér-Rao step fails.

Watch

Extended reading notes

Core claim

The paper's central claim is Eq. (5): after a single step of policy-gradient descent in RLVR, the guaranteed reduction in the loss is bounded by (1/2β) E_x[a_θ(x)(1−a_θ(x))], where a is the current per-instance accuracy. Since a(1−a) is maximized at a=0.5 and vanishes at a=0 or 1, the theory says training data are most informative exactly at the 50% accuracy point, and that a control mechanism able to hold rollout accuracy near 0.5 will train faster. The paper argues that hint scaffolding is such a mechanism: append the first l tokens of a reference solution, and the model's completion accuracy rises monotonically with l; fitting a 3PL IRT curve per instance lets SEELE invert this to find th

Load-bearing premise

The central result assumes that the reward signal behaves like a clean unbiased estimate of per-problem accuracy and that the model's parameter space satisfies a standard statistical regularity condition; for an autoregressive language model these assumptions are not verified, so the 50% optimum is proven only under that premise.

Editorial extensions

If this is right

  • Training data whose rollout accuracy is near 50% yields the largest guaranteed per-step improvement; both trivial and unsolvable problems waste gradient steps.
  • Hint length should be set per instance and per timestep, not by a dataset-wide decay schedule; static schedules lose efficiency as the model's capability changes.
  • The accuracy-hint relationship is S-shaped and fits a 3PL curve, so a few rollout rounds suffice to locate the hint length that targets a chosen accuracy.
  • Because the descent bound is symmetric around 0.5, missing the target high by as much as low hurts equally; both under- and over-guiding are avoidable.
  • SEELE's gains over GRPO/SFT are consistent on both 1.5B and 3B models and transfer to out-of-domain general reasoning, suggesting the sweet-spot principle generalizes across base-model scales.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 50% target is stated per-instance, but the batch-level analogue—keeping the average accuracy of a batch near 0.5—is a testable curriculum that may explain part of the gains; measuring one-step loss descent per accuracy bin would separate the true mechanism from the benefit of added supervised guidance.
  • If the Cramér-Rao replacement fails for autoregressive policies, the theory becomes a heuristic; a direct computation of ∇a^T F^{-1}∇a versus Var(r) on a small transformer would settle whether Eq. (5) is a bound or an approximation.
  • The idea that success probability rather than human-labeled difficulty is the right difficulty signal could transfer to other tunable-effort settings—for example code generation with test feedback or interactive agents—where difficulty can be adjusted at runtime.
  • The S-shaped 'unlocking' of a problem as hint length grows suggests critical steps carry most information; an extension could learn which step unblocks the reasoner and generate hints at the moment they matter, rather than as fixed prefixes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

1 major / 5 minor

Summary. The paper proposes SEELE, a supervision-aided RLVR framework that adaptively adjusts per-instance hint length to keep the policy's rollout accuracy near 50%. The central claim is that learning efficiency, measured by one-step loss descent, is upper-bounded by a quadratic function of accuracy, and therefore maximized at 50% accuracy. To implement this, SEELE uses multi-round rollout sampling, fits a three-parameter logistic (3PL) accuracy-vs-hint-rate model per instance, and predicts the hint length that would hit the 50% target. Experiments on six math and three general reasoning benchmarks with Qwen2.5-1.5B and 3B show consistent improvements over GRPO, SFT, and prior hint-based methods, with an ablation varying the target accuracy supporting the 50% choice.

Significance. If the central claim were rigorously established, it would provide a simple and actionable principle for RLVR: keep rollout accuracy near 50% by revealing just enough of a solution. The proposed framework is novel, operating at instance level with real-time adaptation, and the empirical results are consistent and large in magnitude (+11.8 over GRPO on math average for the 3B model). The paper also includes an extensive set of ablations (target accuracy, rollout scheme, multi-round configuration) and an analysis of the accuracy-hint curves. However, the theoretical derivation in Appendix A does not currently establish the 50% optimum; the main evidence for the central claim is therefore the target-accuracy ablation, which lacks variance reporting. The paper would be strengthened by fixing or reframing the theory and by adding seed-level statistics.

major comments (1)
  1. [Section 4.3, Eq. (12)] The linearity assumption ν = -(k/k0)p + ν0 is introduced without justification. While the empirical curves in Figure 8 and Figure 3 are consistent with the 3PL form, the paper does not report fitting diagnostics (e.g., residuals, goodness-of-fit) or discuss cases where the fitted parameters are non-identifiable. Since the 3PL model is fit using only a small number of points (with margin points added in Algorithm 1), the reliability of the inverse prediction f_φ^{-1}(a*) deserves more scrutiny. Consider adding a brief analysis of fitting stability and error propagation into the predicted hint length.
minor comments (5)
  1. [Abstract and Section 5.2] The abstract states 'outperforms GRPO and SFT by +11.8 and +10.5 points', but the table reports different per-model margins. Clarify whether these numbers are averages over the two model sizes or refer to a specific configuration.
  2. [Algorithm 1] The notation \(\hat{C}_x\) in Algorithm 1 is not defined in the main text; it appears to be the augmented set with margin points. Please define it explicitly.
  3. [Section 5.5] The comment that a three-round scheme has 'only two sample points' is slightly inconsistent with Algorithm 1, which adds margin points (0,0) and (1,1) when not already present. Clarify how the margin points interact with the fitting process.
  4. [Throughout] There is a typo in Section 4.2: 'supervision-aided RVLR' should be 'RLVR'. Also, some reference formats (e.g., 'K ¨opf') have unusual spacing; please use a consistent bibliography style.
  5. [Figure 5] The left and right panels of Figure 5 are not labeled with the model names in the caption text, only in the figure itself. Please add the model names to the caption for clarity.

Circularity Check

0 steps flagged · score 2.0 of 10

No material circularity: Eq. (5) is derived, not fitted, though it is only an upper bound and the 50% optimum is overclaimed.

full rationale

The paper's central claimed result is Eq. (5): L(θold) − L(θold+d) ≤ (1/2β) E_x[a(1−a)]. This is not equivalent to its inputs by construction. The paper defines aθ(x)=E[r], derives the one-step loss descent as (1/2β) ∇a^T F^{-1}∇a in Eq. (42), and then applies a Cramér–Rao bound to replace this with Var(r)=a(1−a) as an upper bound. That is a derivation from a stated model, not a fit. The 50% target is obtained as the maximizer of the quadratic a(1−a), which is a mathematical consequence, not a parameter fitted to benchmark scores. The multi-round 3PL fitting is an online control/estimation procedure for choosing hint lengths; it is not a scientific prediction validated on the same data, and the ablation in §5.4 tests alternative target accuracies rather than selecting the best one after the fact. The only self-citations appear in an introductory related-work sentence (Sun et al., 2025 and Zhang et al., 2025b, both co-authored by Zexu Sun); they are used as general pointers to prior work on data dependence and are not load-bearing for the derivation or the method. There is no uniqueness import and no ansatz smuggled in via self-citation. A correctness caveat remains: Eq. (5) is an upper bound, and the paper's language that learning efficiency itself is maximized at 50% goes beyond what the inequality establishes. That is a validity concern, not a circularity. Hence no significant circularity; the score of 2 reflects the minor self-citations and the overstatement, not a circular reduction.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim rests on (i) the Cramér-Rao/Taylor derivation connecting loss descent to a(1-a), and (ii) a per-instance 3PL fit that is statistically underdetermined at m=4 rounds with n=8 samples. The method also relies on a manually chosen 50% target and several hyperparameters.

free parameters (5)
  • 3PL parameters {k, μ, b} per instance = fitted online, not reported
    Each problem's accuracy-hint curve is fit by nonlinear least squares to rollout accuracies from prior rounds; the fitted curve determines hint length.
  • target accuracy a* = 0.50
    Set from Eq. (5); ablation in §5.4 tests other values and finds 0.5 best. Chosen by hand from the derivation, not fit to validation.
  • multi-round schedule (m, n) = m=4, n=8 (total 32 rollouts)
    Selected empirically in §5.5; four rounds with eight samples outperforms three and more rounds.
  • cold-start hinting rate = (|y|-1)/|y|
    Default near full solution in first round to prepare for worst model capability; then refined. Ad hoc.
  • KL and imitation coefficients = β=0.001, γ=0.001
    Hyperparameters chosen for optimization stability; not derived.
assumptions (4)
  • domain assumption θ_ref = θ_old and local Taylor expansions of L_policy and KL are accurate
    Used to derive the per-instance minimizer d*_x in §4.1 and Appendix A; not true for large neural updates.
  • domain assumption Cramér-Rao bound applies to r(x,y) as an unbiased estimator of a_θ(x) with nonsingular Fisher information
    Needed to replace ∇a^T F^{-1}∇a with Var(r) in Eq. (43); regularity conditions are not checked.
  • ad hoc to paper Problem difficulty is linearly related to hinting rate: ν = -(k/k0)p + ν0
    Imposed in §4.3 to turn the standard 3PL IRT model into Eq. (12); no evidence beyond fitted plots.
  • domain assumption Accuracy increases monotonically in hint length so f_phi is invertible
    The method needs p = f_phi^{-1}(a*) unique; Appendix C shows S-shaped curves on 100 examples, but not guaranteed for all problems.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Staying in the Sweet Spot: Responsive Reasoning Evolution via Capability-Adaptive Hint Scaffolding." pith.science (2026). https://pith.science/paper/MRBW2T5B

@misc{pith2026250906923,
  author       = {Pith},
  title        = {Pith review of: Staying in the Sweet Spot: Responsive Reasoning Evolution via Capability-Adaptive Hint Scaffolding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MRBW2T5B}},
  note         = {Machine review of arXiv:2509.06923}
}
read the original abstract

Reinforcement learning with verifiable rewards (RLVR) has achieved remarkable success in enhancing the reasoning capabilities of large language models (LLMs). However, existing RLVR methods often suffer from exploration inefficiency due to mismatches between the training data's difficulty and the model's capability. LLMs fail to discover viable reasoning paths when problems are overly difficult, while learning little new capability when problems are too simple. In this work, we formalize the impact of problem difficulty by quantifying the relationship between loss descent speed and rollout accuracy. Building on this analysis, we propose SEELE, a novel supervision-aided RLVR framework that dynamically adjusts problem difficulty to stay within the high-efficiency region. SEELE augments each training sample by appending a hint (part of a full solution) after the original problem. Unlike previous hint-based approaches, SEELE deliberately and adaptively adjusts the hint length for each problem to achieve an optimal difficulty. To determine the optimal hint length, SEELE employs a multi-round rollout sampling strategy. In each round, it fits an item response theory model to the accuracy-hint pairs collected in preceding rounds to predict the required hint length for the next round. This instance-level, real-time difficulty adjustment aligns problem difficulty with the evolving model capability, thereby improving exploration efficiency. Experimental results show that SEELE outperforms Group Relative Policy Optimization (GRPO) and Supervised Fine-tuning (SFT) by +11.8 and +10.5 points, respectively, and surpasses the best previous supervision-aided approach by +3.6 points on average across six math reasoning benchmarks.

Figures

Figures reproduced from arXiv: 2509.06923 by the authors.

Figure 1
Figure 1. Comparison between the direct rollout (blue) and hinted rollout (purple). The hint consists [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of SEELE. In each step, SEELE conducts [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Cases of the accuracy-hint curves and the 3PL fitted curve and parameters. We select 4 [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Training dynamics of RL compared with baselines. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Accuracy within each rollout round during training. The red dotted line denotes the [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Performance for different target accuracy [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Reward across the training steps for various target accuracy. [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]
Figure 8
Figure 8. Figure 8: Accuracy with respect to the hinting rate for 100 training examples using Qwen2.5-3B. [PITH_FULL_IMAGE:figures/full_fig_p021_8.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Zone of Proximal Policy Optimization: Teacher in Prompts, Not Gradients

    cs.CL 2026-06 unverdicted novelty 7.0 of 10

    ZPPO improves distillation to small vision-language models by using binary and negative candidate prompts plus a replay buffer for hard questions, outperforming standard distillation and GRPO on a 31-benchmark suite w...

  2. AdaKP: Online Adaptive Knowledge-Point Selection for Reasoning-Oriented Reinforcement Learning

    cs.AI 2026-07 conditional novelty 6.0 of 10

    Online re-selection of atomic hint subsets during RL training improves competition-math accuracy by +1.85 average over a fixed-hint baseline, concentrated on reward-sparse benchmarks.

  3. Off-Context GRPO: Learning to Reason on Hard Problems using Privileged Information

    cs.LG 2026-07 conditional novelty 6.0 of 10

    OC-GRPO reweights GRPO gradients with an importance ratio so that hints used during rollout generation still optimize the original unguided objective, delivering a 13.8% relative Pass@1 gain over vanilla GRPO.

  4. Rollout Pass-Rate Control: Steering Binary-Reward RL Toward Its Most Informative Regime

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    Prefix Sampling replays self-generated trajectory prefixes to control rollout pass rates near 50% in binary-reward RL, delivering wall-clock speedups and modest performance gains on SWE-bench Verified and AIME tasks.

  5. Rollout Pass-Rate Control: Steering Binary-Reward RL Toward Its Most Informative Regime

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    Prefix Sampling steers binary-reward agentic RL rollouts to a 50% pass rate to maximize learning signal, yielding up to 2.01x speedups on SWE-bench with maintained or improved verified performance.

  6. Rollout Pass-Rate Control: Steering Binary-Reward RL Toward Its Most Informative Regime

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    Prefix Sampling replays self-generated trajectory prefixes to control rollout pass rates to ~50% in binary-reward GRPO, delivering 2.01x and 1.55x speedups on Qwen3-14B/32B with slight score improvements on SWE-bench ...

  7. Variance-Aware Baselines and Adaptive Learning Rates for Reinforcement Learning with Verifiable Rewards

    stat.ML 2025-11 reject novelty 4.0 of 10

    OBLR-PO combines an SNR-based learning rate and a gradient-weighted baseline for RLVR, but the main theory is undermined by a flawed smoothness proof.

  8. Variance-Aware Baselines and Adaptive Learning Rates for Reinforcement Learning with Verifiable Rewards

    stat.ML 2025-11 reject novelty 4.0 of 10

    OBLR-PO combines a gradient-weighted leave-one-out baseline with an SNR-adaptive learning-rate schedule for RL with verifiable rewards, and claims improved LLM post-training, though the theoretical proof has a gap.

Reference graph

Works this paper leans on

47 extracted references · 36 canonical work pages · cited by 5 Pith papers

  1. [1]

    Sft or rl? an early investigation into training r1-like reasoning large vision-language models

    Hardy Chen, Haoqin Tu, Fali Wang, Hui Liu, Xianfeng Tang, Xinya Du, Yuyin Zhou, and Cihang Xie. Sft or rl? an early investigation into training r1-like reasoning large vision-language models. arXiv preprint arXiv:2504.11468, 2025

    arXiv 2025

  2. [2]

    Item Response Theory -- A Statistical Framework for Educational and Psychological Measurement , August 2021

    Yunxiao Chen, Xiaoou Li, Jingchen Liu, and Zhiliang Ying. Item Response Theory -- A Statistical Framework for Educational and Psychological Measurement , August 2021

    work page 2021

  3. [3]

    Le, Sergey Levine, and Yi Ma

    Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V. Le, Sergey Levine, and Yi Ma. SFT Memorizes , RL Generalizes : A Comparative Study of Foundation Model Post-training . In Forty-Second International Conference on Machine Learning , June 2025

    work page 2025

  4. [4]

    Think you have Solved Question Answering ? Try ARC , the AI2 Reasoning Challenge , March 2018

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have Solved Question Answering ? Try ARC , the AI2 Reasoning Challenge , March 2018

    work page 2018

  5. [5]

    Training Verifiers to Solve Math Word Problems , November 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training Verifiers to Solve Math Word Problems , November 2021

    work page 2021

  6. [6]

    DeepSeek-AI , Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai ...

    work page 2025

  7. [7]

    Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J

    DeepSeek-AI , Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Hua...

    work page 2025

  8. [8]

    Improving RL Exploration for LLM Reasoning through Retrospective Replay , July 2025

    Shihan Dou, Muling Wu, Jingwen Xu, Rui Zheng, Tao Gui, Qi Zhang, and Xuanjing Huang. Improving RL Exploration for LLM Reasoning through Retrospective Replay , July 2025

    work page 2025

Show all 47 references
  1. [9]

    SRFT : A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoning , June 2025

    Yuqian Fu, Tinghong Chen, Jiajun Chai, Xihuai Wang, Songjun Tu, Guojun Yin, Wei Lin, Qichao Zhang, Yuanheng Zhu, and Dongbin Zhao. SRFT : A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoning , June 2025

    2025

  2. [10]

    Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D. Goodman. Cognitive Behaviors that Enable Self-Improving Reasoners , or, Four Habits of Highly Effective STaRs , March 2025

    2025

  3. [11]

    Navigate the Unknown : Enhancing LLM Reasoning with Intrinsic Motivation Guided Exploration , July 2025

    Jingtong Gao, Ling Pan, Yejing Wang, Rui Zhong, Chi Lu, Qingpeng Cai, Peng Jiang, and Xiangyu Zhao. Navigate the Unknown : Enhancing LLM Reasoning with Intrinsic Motivation Guided Exploration , July 2025

    2025

  4. [12]

    OlympiadBench : A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. OlympiadBench : A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific ...

    2024

  5. [13]

    DeepMath-103K : A Large-Scale , Challenging , Decontaminated , and Verifiable Mathematical Dataset for Advancing Reasoning , May 2025

    Zhiwei He, Tian Liang, Jiahao Xu, Qiuzhi Liu, Xingyu Chen, Yue Wang, Linfeng Song, Dian Yu, Zhenwen Liang, Wenxuan Wang, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. DeepMath-103K : A Large-Scale , Challenging , Decontaminated , and Verifiable Mathematical D...

    2025

  6. [14]

    Measuring Mathematical Problem Solving With the MATH Dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring Mathematical Problem Solving With the MATH Dataset . In Joaquin Vanschoren and Sai-Kit Yeung (eds.), Proceedings of the Neural Information Processing ...

    2021

  7. [15]

    Mathruler

    hiyouga. Mathruler. https://github.com/hiyouga/MathRuler, 2025

    2025

  8. [16]

    Boosting MLLM Reasoning with Text-Debiased Hint-GRPO , June 2025 a

    Qihan Huang, Weilong Dai, Jinlong Liu, Wanggui He, Hao Jiang, Mingli Song, Jingyuan Chen, Chang Yao, and Jie Song. Boosting MLLM Reasoning with Text-Debiased Hint-GRPO , June 2025 a

    2025

  9. [17]

    Ponti, and Ivan Titov

    Zeyu Huang, Tianhao Cheng, Zihan Qiu, Zili Wang, Yinghui Xu, Edoardo M. Ponti, and Ivan Titov. Blending Supervised and Reinforcement Fine-Tuning with Prefix Sampling , July 2025 b

    2025

  10. [18]

    o pf, Yannic Kilcher, Dimitri Von R \

    Andreas K \"o pf, Yannic Kilcher, Dimitri Von R \"u tte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Nguyen, Oliver Stanley, Rich \'a rd Nagyfi, et al. Openassistant conversations-democratizing large language model alignment. Advances in neural info...

    2023

  11. [19]

    Solving Quantitative Reasoning Problems with Language Models

    Aitor Lewkowycz, Anders Johan Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Venkatesh Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo , Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari , and Vedant Misra. Solving Quantitative Reasoning Problems with...

    2022

  12. [20]

    Numinamath

    Jia LI, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. Numinamath. [https://github.com/project-numina/aimo-...

    2024

  13. [21]

    UFT : Unifying Supervised and Reinforcement Fine-Tuning , May 2025 a

    Mingyang Liu, Gabriele Farina, and Asuman Ozdaglar. UFT : Unifying Supervised and Reinforcement Fine-Tuning , May 2025 a

    2025

  14. [22]

    Understanding R1-Zero-Like Training : A Critical Perspective , March 2025 b

    Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding R1-Zero-Like Training : A Critical Perspective , March 2025 b

    2025

  15. [23]

    Lmfit: Non-linear least-squares minimization and curve-fitting for python, July 2025

    Matthew Newville, Renee Otten, Andrew Nelson, Till Stensitzki, Antonino Ingargiola, Daniel Allan, Austin Fox, Faustin Carter, and Michal Rawlik. Lmfit: Non-linear least-squares minimization and curve-fitting for python, July 2025. URL https://doi.org/10.5281/zenodo.16175987

    2025 doi

  16. [24]

    OpenAI, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky , Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennett,...

    2024

  17. [25]

    Qwen2.5 Technical Report , January 2025

    Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le ...

    2025

  18. [26]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA : A Graduate-Level Google-Proof Q & A Benchmark , November 2023

    2023

  19. [27]

    LLMs are Greedy Agents : Effects of RL Fine-tuning on Decision-Making Abilities , April 2025

    Thomas Schmied, J \"o rg Bornschein, Jordi Grau-Moya , Markus Wulfmeier, and Razvan Pascanu. LLMs are Greedy Agents : Effects of RL Fine-tuning on Decision-Making Abilities , April 2025

    2025

  20. [28]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath : Pushing the Limits of Mathematical Reasoning in Open Language Models , April 2024

    2024

  21. [29]

    HybridFlow : A Flexible and Efficient RLHF Framework

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. HybridFlow : A Flexible and Efficient RLHF Framework . In Proceedings of the Twentieth European Conference on Computer Systems , pp.\ 1279--1297, March 2025. doi:...

    2025

  22. [30]

    Uncertainty and influence aware reward model refinement for reinforcement learning from human feedback

    Zexu Sun, Yiju Guo, Yankai Lin, Xu Chen, Qi Qi, Xing Tang, Ji-Rong Wen, et al. Uncertainty and influence aware reward model refinement for reinforcement learning from human feedback. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  23. [31]

    Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, Zhuofu Chen, Jialei Cui, Hao Ding, Mengnan Dong, Angang Du, Chenzhuang Du, Dikang Du, Yulun Du, Yu Fan, Yichen Feng, Kelin Fu, Bofei Gao, Hongcheng Ga...

    2025

  24. [32]

    Reinforcement Learning for Reasoning in Large Language Models with One Training Example , May 2025

    Yiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren, Liyuan Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, Weizhu Chen, Shuohang Wang, Simon Shaolei Du, and Yelong Shen. Reinforcement Learning for Reasoning in Large Language Models with One Training Example , May 2025

    2025

  25. [33]

    MMLU-Pro : A More Robust and Challenging Multi-Task Language Understanding Benchmark

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. MMLU-Pro : A More Robust and Challenging Multi-Task Language Under...

    2024

  26. [34]

    Thought- Augmented Policy Optimization : Bridging External Guidance and Internal Capabilities , May 2025

    Jinyang Wu, Chonghua Liao, Mingkuan Feng, Shuai Zhang, Zhengqi Wen, Pengpeng Shao, Huazhe Xu, and Jianhua Tao. Thought- Augmented Policy Optimization : Bridging External Guidance and Internal Capabilities , May 2025

    2025

  27. [35]

    Learning to Reason under Off-Policy Guidance , May 2025

    Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. Learning to Reason under Off-Policy Guidance , May 2025

    2025

  28. [36]

    DAPO : An Open-Source LLM Reinforcement Learning System at Scale , May 2025

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen...

    2025

  29. [37]

    Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model ?, May 2025

    Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model ?, May 2025

    2025

  30. [38]

    SimpleRL-Zoo : Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild , August 2025

    Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. SimpleRL-Zoo : Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild , August 2025

    2025

  31. [39]

    StepHint : Multi-level Stepwise Hints Enhance Reinforcement Learning to Reason , July 2025 a

    Kaiyi Zhang, Ang Lv, Jinpeng Li, Yongbo Wang, Feng Wang, Haoyuan Hu, and Rui Yan. StepHint : Multi-level Stepwise Hints Enhance Reinforcement Learning to Reason , July 2025 a

    2025

  32. [40]

    A survey on test-time scaling in large language models: What, how, where, and how well? arXiv preprint arXiv:2503.24235, 2025 b

    Qiyuan Zhang, Fuyuan Lyu, Zexu Sun, Lei Wang, Weixu Zhang, Wenyue Hua, Haolun Wu, Zhihan Guo, Yufei Wang, Niklas Muennighoff, et al. A survey on test-time scaling in large language models: What, how, where, and how well? arXiv preprint arXiv:2503.24235, 2025 b

    2025 arXiv

  33. [41]

    On-policy rl meets off-policy experts: Harmonizing supervised fine-tuning and reinforcement learning via dynamic weighting

    Wenhao Zhang, Yuexiang Xie, Yuchang Sun, Yanxi Chen, Guoyin Wang, Yaliang Li, Bolin Ding, and Jingren Zhou. On-policy rl meets off-policy experts: Harmonizing supervised fine-tuning and reinforcement learning via dynamic weighting. arXiv preprint arXiv:2508.11408, 2025 c

    2025

  34. [42]

    Echo Chamber : RL Post-training Amplifies Behaviors Learned in Pretraining , August 2025

    Rosie Zhao, Alexandru Meterez, Sham Kakade, Cengiz Pehlevan, Samy Jelassi, and Eran Malach. Echo Chamber : RL Post-training Amplifies Behaviors Learned in Pretraining , August 2025

    2025

  35. [43]

    Group Sequence Policy Optimization , July 2025

    Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, Jingren Zhou, and Junyang Lin. Group Sequence Policy Optimization , July 2025

    2025

  36. [44]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  37. [45]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  38. [46]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  39. [47]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.