Pith. sign in

REVIEW 3 major objections 4 minor 40 references

A transient low-rank adapter, fitted briefly on ground-truth solutions of prompts where every sampled rollout fails and then discarded, restores the policy-gradient signal that group-relative RL loses on zero-reward 'cliff' prompts.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 01:24 UTC pith:GYXRWSMJ

load-bearing objection The all-16-cell claim can't be checked, the IS correction is biased as conceded, but the transient-LoRA idea is new and worth a serious referee. the 3 major comments →

arxiv 2607.27787 v1 pith:GYXRWSMJ submitted 2026-07-30 cs.LG cs.AI

LoRA Scaffolded Policy Optimization (LSPO): A Sampling-Time Low-Rank Scaffold for Recovering Reinforcement-Learning Gradient on Zero-Reward Cliff Prompts

Ken Ding This is my paper
classification cs.LG cs.AI
keywords reinforcement learning from verifiable rewardsGRPOcliff promptsLoRAimportance samplingmathematical reasoningpolicy optimizationlow-rank adapters
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that the zero-gradient blind spot of group-relative reinforcement learning can be repaired at sampling time without modifying the RL objective. On a cliff prompt—one where every sampled rollout fails—LSPO briefly attaches a low-rank adapter, trains that adapter on the prompt's ground-truth solution, re-samples the prompt with the adapter active, and splices any verifier-passing completions back into the batch before the standard policy update, using an importance-sampling correction so the base model receives an approximately restored gradient. If the claim holds, training would no longer stall on the hardest frontier prompts: the model would learn directly from problems it currently never solves, and the delivered model would remain adapter-free. In the paper's paired five-seed evaluation, the mechanism matches or beats the baseline on all 16 reported benchmark/pass-at-k cells, with an average +3.8-point improvement and roughly 43% of otherwise zero-gradient cliff groups converted into groups carrying usable signal.

Core claim

On the paper's own terms, LSPO recovers the RL gradient on cliff prompts by treating a LoRA adapter as a transient proposal distribution: it fits the adapter by supervised learning on the ground-truth solutions of currently unsolvable prompts, re-rolls those prompts under the base-plus-adapter policy, and splices successful completions into the group-relative batch. The base model receives only the policy-gradient update; the adapter receives only the supervised update and is discarded at checkpoint. The paper's central empirical claim is that this sampling-time scaffold, averaged over five paired seeds at a matched 1000-step reporting horizon, matches or beats the baseline on every reported

What carries the argument

The central object is the transient low-rank adapter (LoRA), a parameter-efficient method that learns a small low-rank weight update while the original weights stay frozen. Here the adapter is fitted for only a few steps on cliff prompts' ground-truth solutions, used solely as the sampling distribution for re-rollouts, and then discarded, so the deliverable is the base model alone. The load-bearing identity is the change-of-measure correction in Eq. (2): the base policy's gradient can be expressed as an expectation under the adapter-augmented policy, reweighting each spliced row by the per-token ratio of base to base-plus-adapter probabilities. This identity is what lets adapter-elicited suc

Load-bearing premise

The load-bearing premise is the change-of-measure identity that lets spliced successes be treated as draws from the adapter-augmented policy, but the implementation actually draws them conditional on passing the verifier, and the paper concedes the resulting update is not an unbiased estimator and carries no convergence guarantee—if that conditional-selection bias is not corrected, the reported gains could stem from a biased heuristic rather than genuine gradient recovery.

What would settle it

Run the same paired-seed comparison with the importance-sampling correction on spliced rows replaced by a fixed weight of 1: if the all-cells win remains largely intact, the outcome comes from selective splicing of successes, not from the claimed change-of-measure correction. A second check is to log the unclipped per-token ratios on spliced rows; if their products routinely fall far outside the asymmetric clip window, the clipped surrogate no longer approximates the base policy gradient.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • The RL loss does not need to be rewritten: the same group-relative surrogate, applied to a batch with spliced successes, produces gradient on prompts that previously contributed exactly zero.
  • The final model is adapter-free; because the scaffold is stripped at every checkpoint, deployment cost and architecture are identical to the base model.
  • The intervention acts at sampling time, so it composes with any group-relative policy-gradient recipe rather than requiring changes to the policy update.
  • The gain is not only a wider sampled distribution: pass-at-1 improves on every benchmark (+1.3 to +4.7 points), indicating a shift in single-sample accuracy.
  • The mechanism's effect is reproducible across seeds: 41.3%–45.5% of cliff groups are converted, so the reported average conversion is not driven by one run.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper treats the importance correction as a change of measure, but its implementation keeps only verifier-passing re-rollouts, so the effective proposal is success-conditional. A natural ablation the paper leaves open: disable the correction and rerun the paired seeds—if the win persists, the mechanism is better described as a biased rescue heuristic than as recovered gradient.
  • The same splice-and-discard recipe could generalize beyond adapters: any cheap privileged proposal that elicits rare successes, such as a different decoding strategy or a short prompted hint, could rescue zero-gradient groups under the same importance-correction logic.
  • Because the paper's own Limitations section notes untested sensitivity to adapter rank, number of supervised steps, splice variant, and answer-space size, a high-value extension is a systematic sweep of those axes; the single-model, single-dataset evaluation leaves transfer open.
  • The reported per-step surpass rate is bimodal (0% on about 31% of cliff-bearing steps and 100% on about 21%), so the aggregate 43% conversion is a mean of extremes; predicting which cliff prompts are surpassable would be a useful diagnostic for deciding when the scaffold will pay for its cost.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes LoRA Scaffolded Policy Optimization (LSPO), a sampling-time wrapper around GRPO/DAPO for RLVR on math reasoning. Each step, LSPO identifies "cliff" prompts whose sampled rollouts all fail, fits a transient LoRA adapter by supervised fine-tuning on those prompts' ground-truth solutions, re-samples the cliffs with the adapter, splices only verifier-passing adapter rollouts back into the RL batch, and applies a GRPO update to the base model alone, with the adapter's sampling density used as the importance-sampling denominator for the spliced rows. The adapter is discarded each step. The paper claims this recovers the otherwise-zero policy gradient on cliffs, and reports that on DeepMath-103K with DeepSeek-R1-Distill-Qwen-1.5B, over n=5 paired seeds at a matched 1000-step horizon, LSPO's mean matches or beats a DAPO baseline on all 16 (benchmark, pass@k) cells, with an average +3.82 point gain. The paper also reports a cliff-conversion rate of about 43%.

Significance. The cliff-prompt problem is real and practically important: standard GRPO indeed assigns zero advantage to groups where all rollouts fail, so the idea of using a transient, sampling-time LoRA scaffold to elicit successful rollouts and then discarding the adapter is elegant and potentially useful. The paper is commendably explicit about its setup, paired-seed design, and limitations, and the cliff-conversion measurement is a good attempt to verify the mechanism directly. However, the central mechanism claim is not established: the importance-sampling correction in Eq. (2) is not valid for the implementation's conditional (verifier-passing) proposal distribution, and the paper itself concedes that the update is not an unbiased policy-gradient estimator and has no convergence guarantee. Consequently, the headline "recovers RL gradient" is unsupported; the reported empirical gains could stem from a biased heuristic. The lack of significance testing further weakens the empirical headline.

major comments (3)
  1. [§3.3, Eq. (2); Algorithm 1 lines 6–7] The change-of-measure identity in Eq. (2) is applied to rows that are not drawn from πμ. Algorithm 1 admits only verifier-passing re-rollouts into S, so the actual proposal is πμ(·|R=1), not πμ. The correct IS weight is πθ(y)/πμ(y|R=1) = Pμ(R=1)·πθ(y)/πμ(y). The implementation uses πθ/πμ and therefore systematically overweights all spliced successes by the missing factor 1/Pμ(R=1). This is not the base policy gradient; it is a biased variant that amplifies adapter successes. The paper's concession in §3.3 that the update is not unbiased and has no convergence guarantee is in direct tension with the abstract and conclusion's claim that LSPO "recovers" the RL gradient. This must be corrected or the claims reframed.
  2. [Table 1 / §4.2] The headline all-16-cell win is based on n=5 means with no paired significance test. AIME sets have only 30 problems, so one problem is 3.33 points; several cells show overlapping seed ranges (e.g., AIME25 pass@1: 26.67–30.00 for both arms) and one cell is an exact tie. Without paired confidence intervals or a test, "15 strict wins" is not established as a reliable effect. Please report per-seed paired deltas, a paired test or effect size, and the distribution of the peak steps across seeds.
  3. [§3.1 and §3.3] The target being optimized is not the original GRPO gradient on cliff prompts. After splicing, group advantages are recomputed on a batch that mixes adapter successes with base failures, changing the advantage values for all rows in the group. Eq. (2) treats A(y) as a fixed function in the change of measure, but the actual update uses advantages that depend on the spliced batch. Thus LSPO optimizes a modified-batch surrogate; the IS ratio corrects only the proposal mismatch, not the replacement of the all-zero advantage. The authors should state this explicitly and justify why this surrogate is desirable.
minor comments (4)
  1. [§2.4] The sentence beginning "The closest of these hints in the context and corrects the importance ratio..." is garbled and should be rewritten for clarity.
  2. [Abstract and throughout] There are several typos, e.g., "onall16" in the abstract and "acliffprompt" in the introduction. Please proofread.
  3. [§4.1] The "first-k-slice" estimator is unbiased for pass@k under exchangeability of the validation rollouts, but this assumption should be stated and justified given that rollouts are generated sequentially. Also, the choice of a 1000-step reporting horizon for runs launched with a 1500-step budget should be justified; reporting sensitivity to the horizon would strengthen the claim.
  4. [Experimental artifacts] No code or data release is mentioned. For an empirical RL paper, providing the training/config code and seed-level logs would substantially aid reproducibility.

Circularity Check

0 steps flagged

No significant circularity: the headline results are external benchmark measurements and the mechanism rests on a standard, admittedly biased, importance-sampling identity.

full rationale

The paper's central claim—LSPO matches or beats DAPO on all 16 (benchmark, pass@k) cells—is an empirical result against external benchmarks (MATH500, AIME24/25/26) with ground-truth answers from an external dataset (DeepMath-103K). The adapter is fitted to ground-truth solutions, but the reported pass@k numbers are measurements of the resulting base-only model, not quantities derived from the fit. The importance-sampling identity in Eq. (2) is a standard change-of-measure identity; the implementation's conditional selection of verifier-passing rows introduces a bias that the paper explicitly concedes in Sec. 3.3 ('we do not claim that its update is an unbiased sequence-level policy-gradient estimator or provide a convergence guarantee'), but that is a correctness/validity concern, not circularity. The 43% cliff-conversion rate is a directly measured mechanism statistic, not a prediction forced by construction. The only self-citation (HDPO, Ding 2026) appears in related-work positioning and does not carry the derivation. No load-bearing step reduces a claimed result to its own inputs, so the derivation is self-contained in the relevant sense.

Axiom & Free-Parameter Ledger

6 free parameters · 7 axioms · 0 invented entities

The method introduces no new physical or model entities; it combines existing building blocks (LoRA, GRPO, IS, SFT). The main unstated costs are the hand-chosen scaffold hyperparameters and the conditional-selection assumption in the IS correction, both of which the empirical claim depends on.

free parameters (6)
  • LoRA rank r / alpha = r=16, alpha=64
    Chosen by hand; controls proposal expressiveness and IS ratio conditioning; no sweep or justification.
  • sft_steps_per_new_cliff = 4
    Chosen by hand; too few steps may not surpass cliffs, too many may push the proposal off-policy; no ablation.
  • SFT learning rate = 2.5e-5
    Chosen for adapter fit; interacts with the 4-step budget.
  • re-rollout count K = 16 (same as base group size)
    Directly determines the group-level surpass rate (43%); smaller K would convert fewer cliffs and change the headline result.
  • splice rule = only verifier-passing re-rollouts replace failed slots
    Design choice that introduces selection bias; the central 'gradient recovery' claim depends on this rule.
  • DAPO recipe hyperparameters = eps_low=0.20, eps_high=0.28, lr=1e-6, 32 prompts/step, 16 rollouts/prompt, overlong shaping past 7168 tokens
    Fixed identical to the baseline; not fitted in this paper, but the empirical claim is conditional on this configuration.
axioms (7)
  • standard math GRPO group-normalized advantage is zero when the group reward sum is zero (Eq. 1, Sec 2.2).
    Standard GRPO math; establishes the cliff problem that motivates the method.
  • domain assumption Ground-truth solutions in DeepMath-103K are correct, complete worked traces usable as SFT targets.
    The scaffold's supervised step (Alg 1 line 5) trains on these traces; if they are wrong or unrepresentative, the proposal fails.
  • ad hoc to paper Eq. (2) change of measure remains valid for the implementation that splices only verifier-passing rows.
    The actual sampling density is pi_mu(·|R=1), not pi_mu, so Eq. (2) does not strictly apply; authors concede the update is not unbiased (Sec 3.3).
  • domain assumption Verifier reward R is deterministic, correct, and binary for math answers.
    Used to define cliffs and to select spliced rows; noisy verification would corrupt cliff detection and splicing.
  • domain assumption DeepMath-103K is decontaminated with respect to MATH500 and AIME24/25/26.
    Cited to He et al. 2025; benchmark gains would be inflated by leakage if this fails.
  • standard math First-k-slice pass@k estimator is unbiased under exchangeability of the 16 validation rollouts.
    The estimator ignores subset averaging but is unbiased; higher variance, applied identically to both arms.
  • ad hoc to paper Four SFT steps on a rank-16 LoRA keeps pi_theta+phi close enough to pi_theta for the clipped IS ratio to be well-conditioned.
    Variance motivation, not an exactness guarantee; no diagnostics of ratio magnitudes are reported.

pith-pipeline@v1.3.0-daily-deepseek · 10916 in / 12810 out tokens · 111912 ms · 2026-08-01T01:24:01.945838+00:00 · methodology

0 comments
read the original abstract

Reinforcement learning from verifiable rewards (RLVR) for mathematical reasoning suffers from a structural blind spot: on "cliff" prompts-those on which every sampled rollout in a group fails-the group-normalized advantage is identically zero, so GRPO produces no gradient on precisely the prompts at the frontier of the model's capability. We introduce LoRA Scaffolded Policy Optimization (LSPO), a sampling-time mechanism that recovers this lost gradient. Each RL step, LSPO detects cliff prompts, fits a small low-rank (LoRA) adapter by a brief supervised step on their ground-truth solutions, re-rolls the cliffs with the base-plus-adapter model, splices the now-successful completions back into the RL batch with an importance-sampling correction, and takes a GRPO step on the base alone; the adapter receives only the supervised gradient and is discarded at checkpoint, yielding a base-only model. On DeepMath-103K with DeepSeek-R1-Distill-Qwen-1.5B, evaluated over n=5 paired seeds per arm at a matched 1000-step reporting horizon, LSPO's 5-seed mean matches or beats a DAPO baseline on all 16 (benchmark, pass@k) cells (15 strict wins and one exact tie), with gains of up to +10.7 points on AIME24/pass@4, +6.7 points on AIME24 and AIME26 at pass@16, and +2.4 points on MATH500/pass@1; averaged over the 16 cells the improvement is +3.8 points.

Figures

Figures reproduced from arXiv: 2607.27787 by Ken Ding.

Figure 1
Figure 1. Figure 1: The LSPO cliff-rescue mechanism. (1) On a cliff prompt p, all K rollouts from the base policy π fail (red); the group reward sum is zero and standard GRPO has no learning signal. (2) A trainable LoRA adapter ϕLoRA is composed with the (frozen) base π and fitted via SFT on the dataset’s ground-truth trajectory T. (3) Under the composed policy π + ϕLoRA, some rollouts on the same prompt now pass the verifier… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

40 extracted references · 13 linked inside Pith

  1. [1]

    Tina: Tiny Reasoning Models via

    Wang, Shangshang and Asilis, Julian and Akg. Tina: Tiny Reasoning Models via. arXiv preprint arXiv:2504.15777 , year =

    Pith/arXiv arXiv

  2. [2]

    arXiv preprint arXiv:2506.07527 , year =

    Ma, Lu and Liang, Hao and Qiang, Meiyi and Tang, Lexiang and Ma, Xiaochen and Wong, Zhen Hao and Niu, Junbo and Shen, Chengyu and He, Runming and Li, Yanhao and Cui, Bin and Zhang, Wentao , title =. arXiv preprint arXiv:2506.07527 , year =

    arXiv

  3. [3]

    arXiv preprint arXiv:2509.06948 , year =

    Chen, Liang and Han, Xueting and Shen, Li and Bai, Jing and Wong, Kam-Fai , title =. arXiv preprint arXiv:2509.06948 , year =

    Pith/arXiv arXiv

  4. [4]

    arXiv preprint arXiv:2603.23871 , year =

    Ding, Ken , title =. arXiv preprint arXiv:2603.23871 , year =

    arXiv

  5. [5]

    and Wu, Y

    Shao, Zhihong and Wang, Peiyi and Zhu, Qihao and Xu, Runxin and Song, Junxiao and Bi, Xiao and Zhang, Haowei and Zhang, Mingchuan and Li, Y.K. and Wu, Y. and Guo, Daya , title =. arXiv preprint arXiv:2402.03300 , year =

    Pith/arXiv arXiv

  6. [6]

    arXiv preprint arXiv:2503.14476 , year =

    Yu, Qiying and Zhang, Zheng and Zhu, Ruofei and Yuan, Yufeng and others , title =. arXiv preprint arXiv:2503.14476 , year =

    Pith/arXiv arXiv

  7. [7]

    arXiv preprint arXiv:1707.06347 , year =

    Schulman, John and Wolski, Filip and Dhariwal, Prafulla and Radford, Alec and Klimov, Oleg , title =. arXiv preprint arXiv:1707.06347 , year =

    Pith/arXiv arXiv

  8. [8]

    and Shen, Yelong and Wallis, Phillip and Allen-Zhu, Zeyuan and Li, Yuanzhi and Wang, Shean and Wang, Lu and Chen, Weizhu , title =

    Hu, Edward J. and Shen, Yelong and Wallis, Phillip and Allen-Zhu, Zeyuan and Li, Yuanzhi and Wang, Shean and Wang, Lu and Chen, Weizhu , title =. arXiv preprint arXiv:2106.09685 , year =

    Pith/arXiv arXiv

  9. [9]

    arXiv preprint arXiv:2501.12948 , year =

    Pith/arXiv arXiv

  10. [10]

    arXiv preprint arXiv:2504.11456 , year =

    He, Zhiwei and Liang, Tian and Xu, Jiahao and Liu, Qiuzhi and Chen, Xingyu and Wang, Yue and Song, Linfeng and Yu, Dian and Liang, Zhenwen and Wang, Wenxuan and Zhang, Zhuosheng and Wang, Rui and Tu, Zhaopeng and Mi, Haitao and Yu, Dong , title =. arXiv preprint arXiv:2504.11456 , year =

    Pith/arXiv arXiv

  11. [11]

    Neural Information Processing Systems , year =

    Hendrycks, Dan and Burns, Collin and Kadavath, Saurav and Arora, Akul and Basart, Steven and Tang, Eric and Song, Dawn and Steinhardt, Jacob , title =. Neural Information Processing Systems , year =

  12. [12]

    International Conference on Learning Representations , year =

    Lightman, Hunter and Kosaraju, Vineet and Burda, Yura and Edwards, Harri and Baker, Bowen and Lee, Teddy and Leike, Jan and Schulman, John and Sutskever, Ilya and Cobbe, Karl , title =. International Conference on Learning Representations , year =

  13. [13]

    International Conference on Learning Representations , year =

    Loshchilov, Ilya and Hutter, Frank , title =. International Conference on Learning Representations , year =

  14. [14]

    arXiv preprint arXiv:2107.03374 , year=

    Evaluating Large Language Models Trained on Code , author=. arXiv preprint arXiv:2107.03374 , year=

    Pith/arXiv arXiv

  15. [15]

    Understanding

    Liu, Zichen and Chen, Changyu and Li, Wenjun and Qi, Penghui and Pang, Tianyu and Du, Chao and Lee, Wee Sun and Lin, Min , journal =. Understanding. 2025 , url =

    2025

  16. [16]

    arXiv preprint arXiv:2506.13923 , year =

    Adaptive Guidance Accelerates Reinforcement Learning of Reasoning Models , author =. arXiv preprint arXiv:2506.13923 , year =

    Pith/arXiv arXiv

  17. [17]

    2026 , note =

    Zhang, Xichen and Wu, Sitong and Zhu, Yinghao and Tan, Haoru and Yu, Shaozuo and He, Ziyi and Jia, Jiaya , booktitle =. 2026 , note =

    2026

  18. [18]

    arXiv preprint arXiv:2602.03143 , year =

    Self-Hinting Language Models Enhance Reinforcement Learning , author =. arXiv preprint arXiv:2602.03143 , year =

    arXiv

  19. [19]

    arXiv preprint arXiv:2604.00698 , year =

    Learning to Hint for Reinforcement Learning , author =. arXiv preprint arXiv:2604.00698 , year =

    arXiv

  20. [20]

    Nudging the Boundaries of

    Chen, Justin Chih-Yao and Peng, Becky Xiangyu and Choubey, Prafulla Kumar and Huang, Kung-Hsiang and Zhang, Jiaxin and Bansal, Mohit and Wu, Chien-Sheng , booktitle =. Nudging the Boundaries of. 2026 , note =

    2026

  21. [21]

    2025 , url =

    Liu, Ziru and Gong, Cheng and Fu, Xinyu and Liu, Yaofang and Chen, Ran and Hu, Shoubo and Zhang, Suiyun and Liu, Rui and Zhang, Qingfu and Tu, Dandan , journal =. 2025 , url =

    2025

  22. [22]

    arXiv preprint arXiv:2606.18216 , year =

    Zone of Proximal Policy Optimization: Teacher in Prompts, Not Gradients , author =. arXiv preprint arXiv:2606.18216 , year =

    Pith/arXiv arXiv

  23. [23]

    2025 , note =

    Zhang, Xuechen and Huang, Zijian and Li, Yingcong and Ni, Chenshun and Chen, Jiasi and Oymak, Samet , booktitle =. 2025 , note =

    2025

  24. [24]

    2026 , url =

    Guo, Yiqiu and Han, Xueting and Jia, Qi and Zhai, Guangtao and Bai, Jing , journal =. 2026 , url =

    2026

  25. [25]

    and Jeon, Myeongho and Vu, Kim and Lai, Viet and Yang, Eunho , booktitle =

    Le, Thanh-Long V. and Jeon, Myeongho and Vu, Kim and Lai, Viet and Yang, Eunho , booktitle =. No Prompt Left Behind: Exploiting Zero-Variance Prompts in. 2026 , note =

    2026

  26. [26]

    Don't Waste Mistakes: Leveraging Negative

    Feng, Yunzhen and Jain, Parag and Hartshorn, Anthony and Duan, Yaqi and Kempe, Julia , journal =. Don't Waste Mistakes: Leveraging Negative. 2025 , url =

    2025

  27. [27]

    2026 , url =

    Mao, Yixiu and Qu, Yun and Wang, Qi and Zou, Heming and Ji, Xiangyang , journal =. 2026 , url =

    2026

  28. [28]

    Learning-Zone Energy: Online Data Selection for Efficient

    Cui, Peng and Yang, Boyao and Zhu, Jun , journal =. Learning-Zone Energy: Online Data Selection for Efficient. 2026 , url =

    2026

  29. [29]

    Advances in Neural Information Processing Systems (

    Thinking Fast and Slow with Deep Learning and Tree Search , author =. Advances in Neural Information Processing Systems (. 2017 , note =

    2017

  30. [30]

    , booktitle =

    Zelikman, Eric and Wu, Yuhuai and Mu, Jesse and Goodman, Noah D. , booktitle =. 2022 , note =

    2022

  31. [31]

    arXiv preprint arXiv:2308.01825 , year =

    Scaling Relationship on Learning Mathematical Reasoning with Large Language Models , author =. arXiv preprint arXiv:2308.01825 , year =

    Pith/arXiv arXiv

  32. [32]

    Reinforced Self-Training (

    Gulcehre, Caglar and Le Paine, Tom and Srinivasan, Srivatsan and Konyushkova, Ksenia and Weerts, Lotte and Sharma, Abhishek and Siddhant, Aditya and Ahern, Alex and Wang, Miaosen and Gu, Chenjie and Macherey, Wolfgang and Doucet, Arnaud and Firat, Orhan and de Freitas, Nando , journal =. Reinforced Self-Training (. 2023 , url =

    2023

  33. [33]

    Transactions on Machine Learning Research (

    Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models , author =. Transactions on Machine Learning Research (. 2024 , note =

    2024

  34. [34]

    Learn Hard Problems During

    Wu, Yangzhen and Li, Shanda and Wen, Zixin and Zhou, Xin and Talwalkar, Ameet and Yang, Yiming and Huang, Wenhao and Cai, Tianle , journal =. Learn Hard Problems During. 2026 , url =

    2026

  35. [35]

    Robotics: Science and Systems (

    Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations , author =. Robotics: Science and Systems (. 2018 , note =

    2018

  36. [36]

    Advances in Neural Information Processing Systems (

    Learning to Reason under Off-Policy Guidance , author =. Advances in Neural Information Processing Systems (. 2025 , note =

    2025

  37. [37]

    2024 , note =

    Wang, Peiyi and Li, Lei and Shao, Zhihong and Xu, Runxin and Dai, Damai and Li, Yifei and Chen, Deli and Wu, Yu and Sui, Zhifang , booktitle =. 2024 , note =

    2024

  38. [38]

    arXiv preprint arXiv:2502.01456 , year =

    Process Reinforcement through Implicit Rewards , author =. arXiv preprint arXiv:2502.01456 , year =

    Pith/arXiv arXiv

  39. [39]

    Efficient

    Santacroce, Michael and Lu, Yadong and Yu, Han and Li, Yuanzhi and Shen, Yelong , journal =. Efficient. 2023 , url =

    2023

  40. [40]

    International Conference on Machine Learning (ICML) , year =

    Kim, Hyunjik and Papamakarios, George and Mnih, Andriy , title =. International Conference on Machine Learning (ICML) , year =