Pith. sign in

REVIEW 4 major objections 5 minor 67 references

A per-token importance ratio lets RL training use guided rollouts while optimizing the unguided deployment objective, with 13.8% relative Pass@1 gains over vanilla GRPO on math benchmarks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 12:49 UTC pith:M4WYYRWG

load-bearing objection The off-context problem is real and the empirical study is careful, but the paper's headline unbiasedness guarantee applies to a response-level estimator, not the per-token surrogate that is actually trained. the 4 major comments →

arxiv 2607.19313 v1 pith:M4WYYRWG submitted 2026-07-21 cs.LG cs.AI

Off-Context GRPO: Learning to Reason on Hard Problems using Privileged Information

Priyank Agrawal , Ankur Samanta , Shervin Ghasemlou , Jalaj Bhandari , Kavosh Asadi , Daniel Jiang , Aditya Modi This is my paper
classification cs.LG cs.AI
keywords RLVRGRPOoff-context correctionimportance samplinglearning cliffprivileged guidancecredit assignmentmath reasoning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Hard reasoning problems break reinforcement learning with verifiable rewards: when every sampled answer is wrong, group-relative rewards collapse to zero and the gradient vanishes, a learning cliff the paper names and targets. The paper argues that sampling rollouts from a prompt augmented with privileged guidance, such as a solution prefix, breaks the cliff, and that the resulting off-context mismatch can be corrected by reweighting each token by the ratio of the unguided-policy probability to the guided-policy probability. This makes the training gradient unbiased for the objective used at deployment rather than the misaligned objective optimized by guided-target baselines. The same reweighting acts as behavior-aware credit assignment: successes that depend heavily on the hint are damped, failures that persist despite guidance are amplified, so the model is pushed to solve problems rather than copy hints. On hard math problems where the base model never succeeds, the method improves average Pass@1 over vanilla GRPO by 13.8% relative at 7B scale and keeps gains at smaller scales where guided baselines degrade below vanilla.

Core claim

The central claim is that you do not have to choose between breaking the learning cliff and aligning the training objective. Off-Context GRPO samples rollouts from a guided prompt g(x), a solution prefix or hint present only during training, but reweights every token's advantage by the ratio of the unguided-policy probability to the guided-policy probability. The reweighted expectation equals the unguided objective, so the estimator is unbiased for the deployment objective while still obtaining non-zero rewards on problems where unguided sampling gives none. The paper further shows this reweighting is a behavior-aware credit-assignment rule: guided successes receive damped credit, guided fai

What carries the argument

The off-context importance ratio is the load-bearing object: ρ^oc_{i,t}(θ) = π_θ(y_{i,t} | x, y_{i,<t}) / π_{θ_old}(y_{i,t} | g(x), y_{i,<t}), where g(x) is the prompted privileged guidance. Replacing the standard GRPO per-token ratio with this ratio converts guided rollouts into unbiased estimates of the unguided objective. The variance analysis splits the ratio into the standard drift factor and the off-context factor γ_{i,t}, whose Rényi-2 divergence bound e^{2nη} − 1 depends only on guidance length n under the paper's per-guidance-token regularity assumption.

Load-bearing premise

Everything practical rests on Assumption 1—that adding a single guidance token moves the old policy's next-token log-probability by at most a small η—because the response-level importance weight is a product over the full rollout and can scale like e^{T·nη}; if η is not tiny the unbiased estimator's variance explodes, and the paper does not measure η on real models.

What would settle it

Take a trained checkpoint and a set of hard problems with prefix guidance. For each guidance token k and each sampled continuation token y, compute log π_θ_old(y | x,u_{1:k−1},·) − log π_θ_old(y | x,u_{1:k},·); if the max absolute shift is not small (say η > 0.05) or the empirical variance of the response-level importance weight over guided rollouts grows exponentially in rollout length, the low-variance claim is empirically vacuous. A complementary check: compare training runs with and without clipping on the off-context factor; if they diverge in reward, clipping is masking estimator bias.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Hard problems with zero unguided success become trainable: guided rollouts carry verifier reward, and the importance correction keeps the gradient pointing at the deployment objective.
  • The correction implements behavior-aware credit assignment without reward shaping: a success is credited only to the extent the unguided policy could have produced it, while a failure that persists under guidance draws a larger penalty.
  • The design principle 'use the shortest guidance that breaks the cliff' follows from the variance bound, because correction variance scales with guidance token count n, not rollout length T.
  • The mechanism is context-agnostic, so any training-time privileged signal—solution prefixes, hints, tool results, oracle subgoals—can be corrected the same way, extending to multi-turn or agentic RLVR.
  • At smaller model scales the correction matters more: guided-target baselines drop below vanilla GRPO while OC-GRPO retains positive relative gains, suggesting back-generalization is capacity-dependent.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if the per-token regularity assumption holds on real models, one could adaptively grow the guidance prefix during training without retuning, because the correction automatically discounts the extra hint; this suggests a smoother curriculum than the fixed-level schedule the paper defaults to.
  • Editorial inference: the unbiasedness theorem is response-level; the implemented token-level clipped surrogate is not exactly unbiased, so a direct test of the method's honesty is to measure the bias introduced by clipping on a held-out batch.
  • Editorial inference: the same off-context correction could be used for counterfactual credit assignment in hindsight relabeling—change the sampling prompt first and correct the gradient back, instead of relabeling goals to match trajectories, which may reduce distributional distortion.
  • Editorial inference: if guidance selection is applied periodically with the current policy, the variance bound suggests the required guidance length shrinks as the policy improves, so per-problem training cost should decrease over time; this is testable by logging guidance-level distributions across epochs.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces OC-GRPO, a modification of GRPO that uses rollouts generated from a guided prompt g(x) while optimizing the original unguided objective J(θ). The key idea is to reweight per-token advantages by an importance ratio πθ(y_t|x,y_<t) / πθ_old(y_t|g(x),y_<t), with claims that this yields an unbiased estimator of J(θ), variance that scales with the guidance length rather than the rollout length, and a behavior-aware credit-assignment mechanism. The empirical section compares OC-GRPO against vanilla GRPO and three guided baselines on hard MATH problems with Qwen2.5 models, reporting a 13.8% relative Pass@1 improvement over vanilla GRPO at 7B scale and consistent gains at 3B and 1.5B.

Significance. If the theoretical claims held for the implemented algorithm, OC-GRPO would be a practically valuable, minimally invasive way to escape the zero-reward learning cliff on hard reasoning problems: guided rollouts would provide signal while importance correction would provably keep training aligned with the deployment objective. The paper is strong on setup: a clear problem formulation, a fair and consistently implemented comparison protocol, three seeds at 7B, held-out benchmarks, and code release. The algebraic proofs of Lemma 1 and Theorems 2–4 are correct under their stated assumptions. However, the central unbiasedness guarantee is proved only for a response-level estimator, while the implemented loss uses a per-token surrogate whose gradient is generally biased; the variance guarantee is also not established for the quantities the algorithm actually uses. These are load-bearing gaps between theory and practice, so the paper requires major revision.

major comments (4)
  1. [Section 4, Eq. (9) vs. Appendix E.1, Theorem 3] Theorem 3 proves unbiasedness for the response-level estimator Ĝ = [πθ(y|x)/πθ_old(y|g(x))] r ∇log πθ(y|x). The implemented loss (9) instead uses per-token ratios ρ^oc_{i,t} and averages them over T. At θ=θ_old, the gradient of (9) is E_Q[Â_i (1/T_i) Σ_t γ_{i,t} s_{i,t}], while the unbiased response-level gradient is E_Q[Â_i (∏_t γ_{i,t}) Σ_t s_{i,t}]. These differ: e.g., T=2, γ_1=2, γ_2=0.5 gives 0.5 s_1 + 0.25 s_2 vs. s_1 + s_2. Thus the Abstract and Section 1 claim that OC-GRPO is unbiased for J(θ) does not hold for the evaluated per-token surrogate; clipping further adds bias. The authors should either implement the response-level correction or provide a formal bound on the bias of the per-token surrogate and revise the claims accordingly.
  2. [Appendix E.2, Theorem 4 and Assumption 1] Theorem 4 bounds the per-token off-context factor γ_{i,t}, not the response-level importance weight ∏_t γ_{i,t} used in the unbiased estimator, nor the per-token sum (1/T_i)Σ_t γ_{i,t}. Under Assumption 1 the response-level weight can scale as e^{Tnη}, so the statement that variance 'scales with the length of the guidance prefix rather than the rollout' is unsupported. Moreover, Assumption 1's η is never measured on the actual models, so even the per-token bound is unverified. The paper should either measure η empirically, derive a response-level variance bound, or substantially soften the variance claim.
  3. [Section 5, Theorem 2] Theorem 2's credit-assignment decomposition is proved for the response-level surrogate L^resp_OC-GRPO, which uses a centered advantage without group normalization and no clipping. It does not directly describe the per-token, clipped, group-normalized objective in Eq. (9). The paper presents Theorem 2 as the formalization of the OC-GRPO intuition, but the connection to the implemented algorithm is not established. A statement clarifying that Theorem 2 describes an idealized version of the update would be acceptable; without it, the behavior-aware credit assignment claim overstates what follows from the theory.
  4. [Tables 3 and 4] The claims that guided-target baselines 'degrade below vanilla GRPO at smaller scales' and that OC-GRPO retains 'consistent gains' are based on single-seed runs at 3B and 1.5B. Differences such as +7.2% vs. -3.4% are within plausible seed noise, and no error bars are given. These scaling conclusions should be downgraded to preliminary observations or supported with additional seeds.
minor comments (5)
  1. [Eq. (10)] The underbrace annotations for 'success weight' and 'failure weight' appear misaligned with the factors they label; please check the notation.
  2. [Section 6.2 / Appendix G] The evaluation selects the fourth-epoch checkpoint without describing a validation-based selection criterion, even though the text mentions evaluating checkpoints at each epoch. Please clarify whether the fourth epoch is fixed a priori.
  3. [Appendix G] The claim of 'negligible additional cost' applies to OC-GRPO-Fixed, but OC-GRPO-Adaptive incurs repeated inference at each training step. The paper should state this distinction explicitly near the abstract or conclusion.
  4. [Assumption 1] The statement that η is 'small' for a well-calibrated language model is a heuristic. If the authors keep Assumption 1, they should provide at least one empirical histogram of the per-token log-probability shifts on a held-out set.
  5. [Table 2 caption] Bolding based on mean alone with no adjustment for seed variance is acceptable only if the caption says so; consider adding an explicit caveat that differences may not be statistically significant.

Circularity Check

0 steps flagged

No significant circularity: OC-GRPO is a direct application of standard importance sampling; the theory/implementation mismatch is a correctness risk, not a circular derivation.

full rationale

The paper's load-bearing theoretical claims derive from the standard change-of-measure identity: Theorem 3 proves response-level unbiasedness by direct substitution, E_{y~π_old(·|g(x))}[ (π_θ(y|x)/π_old(y|g(x))) r ∇log π_θ(y|x) ] = ∇_θ J(θ;x). This is an algebraic identity, not a restatement of the empirical results or a fitted quantity relabeled as a prediction. Theorem 2's 'behavior-aware credit assignment' is likewise a consequence of the definitions of p(x), q(x), λ_+, and λ_−; it does not assume the conclusion it derives. Assumption 1 and Theorem 4 are explicit assumptions and per-token bounds, and while the paper overstates their applicability to the response-level product, that is a mathematical-variance gap rather than circular reasoning. The references for importance sampling are external and standard (Precup et al., 2001; Degris et al., 2012), and there is no load-bearing self-citation chain or imported uniqueness claim. The empirical gains are measured against held-out benchmarks with baselines reimplemented in the same harness, so they are not built into the derivation. The main discrepancy—response-level unbiasedness versus the clipped per-token surrogate in Eq. (9)—is a real internal-validity concern about whether the implemented estimator matches the proved estimator, but it is not an instance of a derivation reducing to its own inputs. The limitations section also candidly notes scale, domain, and guidance-type restrictions. Under the stated circularity criteria, no circular step is present.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The theoretical core needs only the change-of-measure identity plus Assumption 1; the empirical margin depends on the disclosed hand-chosen filter, prefix grid, level selector, and standard hyperparameters, none of which are fitted to the evaluation benchmarks. No new entities (forces, particles, latent variables) are postulated; 'off-context' is a descriptive label for the g(x)-versus-x distribution mismatch, not a mechanism with separate falsifiable handles.

free parameters (5)
  • hard-problem filter = 64 rollouts under πref, all-fail retained (595 problems at 7B)
    Hand-chosen threshold defining the training distribution for every method; with 64 samples the 95% CI upper bound on p(x) is ~4.5%, so 'hard' does not guarantee p(x)≈0 — a load-bearing choice for the learning-cliff premise.
  • guidance fraction grid = α ∈ {0.2, 0.4, 0.6, 0.8, 1.0}
    Hand-chosen prefix levels (Eq. 11) converted to prefixes by character fraction snapped to word boundaries; the grid density and the 'minimum level' selection rule are arbitrary but disclosed.
  • guidance-level selector = minimal ℓ with ≥1 success in 8 base-model rollouts
    Per-problem choice of ℓ*(x) fixed before training using πref; the 8-rollout budget is hand-chosen and affects how often guidance is strong enough to create a non-degenerate group.
  • RLVR hyperparameters = LoRA r=64, α=128; lr 1e-5; batch 32; N=16; clip ε=0.2; β=0; T=0.7; top-p 0.95
    Standard RLVR hyperparameters taken from cited practice and identical across all methods; not fitted to the eval benchmarks, but they co-determine the reported margins.
  • advantage normalization convention = advantages zeroed when group σ_r = 0
    Standard GRPO convention (Section 2); can zero OC-GRPO's signal on all-success guided groups, a degenerate regime the Section 5 theory (which uses q(x)-centered advantages) does not analyze.
axioms (5)
  • standard math Standard change-of-measure identity E_Q[(P/Q)f] = E_P[f], with support condition Q>0 on the support of P·r
    Used in Section 4 and Theorem 3; requires πθold(y|g(x)) > 0 whenever πθ(y|x)r(x,y) > 0.
  • domain assumption Assumption 1 — per-guidance-token log-prob shift bounded by η, η small
    Load-bearing for Theorem 4 variance control (Appendix E.2); asserted to hold for 'well-calibrated' LMs, with no empirical verification in the paper.
  • domain assumption Verifier reward r(x,y)=1{a(y)=a*(x)} correctly identifies correctness via answer extraction and symbolic equivalence
    Section 2; noisy extraction or equivalence errors would corrupt both guidance-level selection and the target objective.
  • domain assumption MATH reference solutions z*(x) are correct and usable as prefixes
    Eq. (11) builds all guidance from z*(x); incorrect or non-atomic reference solutions would weaken the cliff-escape guarantee.
  • domain assumption Zero-advantage convention when σ_r = 0
    Section 2 convention; the Section 5 theory instead uses q(x)-centered advantages, so theory and implementation diverge on degenerate all-success/all-failure groups.

pith-pipeline@v1.3.0-alltime-deepseek · 23534 in / 31973 out tokens · 302379 ms · 2026-08-01T12:49:03.595466+00:00 · methodology

0 comments
read the original abstract

Reinforcement learning with verifiable rewards (RLVR) improves reasoning in large language models. Yet, typical RLVR approaches fail on difficult problems: when a model cannot generate any correct solutions, it receives \textit{zero} learning signal. Providing privileged guidance during training, such as solution prefixes, can help overcome this learning cliff by steering the model towards {correct solutions with non-zero reward}. {We call these rollouts \textit{off-context}: they are generated from a training prompt that contains privileged guidance, while the target objective is defined by the original prompt without that guidance.} {We introduce} Off-Context GRPO (OC-GRPO), a minimally modified variant of GRPO that uses guided rollouts but applies an importance-corrected objective to steer the update back toward the original unguided objective, avoiding the mismatch that destabilizes uncorrected guided training. Empirically, our algorithm achieves a 3.9\% absolute improvement (13.8\% relative gain) over vanilla GRPO on average across standard mathematical reasoning benchmarks with negligible additional cost.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

67 extracted references · 26 linked inside Pith

  1. [1]

    arXiv preprint arXiv:2508.17445 , year=

    Treepo: Bridging the gap of policy optimization and efficacy and inference efficiency with heuristic tree-based modeling , author=. arXiv preprint arXiv:2508.17445 , year=

    Pith/arXiv arXiv

  2. [4]

    Proceedings of the nineteenth international conference on machine learning , pages=

    Approximately optimal approximate reinforcement learning , author=. Proceedings of the nineteenth international conference on machine learning , pages=

  3. [7]

    ICML , pages=

    Off-policy temporal-difference learning with function approximation , author=. ICML , pages=

  4. [9]

    Edward J Hu and yelong shen and Phillip Wallis and Zeyuan Allen-Zhu and Yuanzhi Li and Shean Wang and Lu Wang and Weizhu Chen , booktitle=. Lo

  5. [10]

    Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles , year=

    Efficient Memory Management for Large Language Model Serving with PagedAttention , author=. Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles , year=

  6. [11]

    2024 , journal =

    HybridFlow: A Flexible and Efficient RLHF Framework , author =. 2024 , journal =

    2024

  7. [13]

    Qwen2.5: A Party of Foundation Models , url =

    Qwen , month =. Qwen2.5: A Party of Foundation Models , url =

  8. [14]

    2023 , publisher =

    Hemish Veeraboina , title =. 2023 , publisher =

    2023

  9. [15]

    MathArena: Evaluating LLMs on Uncontaminated Math Competitions , author =

  10. [18]

    NeurIPS , year=

    Measuring Mathematical Problem Solving With the MATH Dataset , author=. NeurIPS , year=

  11. [21]

    arXiv preprint arXiv:2509.21154 , year=

    GRPO is Secretly a Process Reward Model , author=. arXiv preprint arXiv:2509.21154 , year=

    Pith/arXiv arXiv

  12. [22]

    arXiv preprint arXiv:2510.02263 , year=

    RLAD: Training LLMs to Discover Abstractions for Solving Reasoning Problems , author=. arXiv preprint arXiv:2510.02263 , year=

    arXiv

  13. [23]

    arXiv preprint arXiv:2510.01833 , year=

    Plan Then Action: High-Level Planning Guidance Reinforcement Learning for LLM Reasoning , author=. arXiv preprint arXiv:2510.01833 , year=

    Pith/arXiv arXiv

  14. [25]

    arXiv preprint arXiv:2507.06892 , year=

    Squeeze the soaked sponge: Efficient off-policy reinforcement finetuning for large language model , author=. arXiv preprint arXiv:2507.06892 , year=

    Pith/arXiv arXiv

  15. [28]

    International Conference on Machine Learning , pages=

    Learning to search better than your teacher , author=. International Conference on Machine Learning , pages=. 2015 , organization=

    2015

  16. [33]

    Advances in neural information processing systems , volume=

    Hindsight experience replay , author=. Advances in neural information processing systems , volume=

  17. [36]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Adacurl: Adaptive curriculum reinforcement learning with invalid sample mitigation and historical revisiting , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  18. [42]

    Advances in Neural Information Processing Systems , volume=

    Star: Bootstrapping reasoning with reasoning , author=. Advances in Neural Information Processing Systems , volume=

  19. [48]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Trial and error: Exploration-based trajectory optimization of LLM agents , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  20. [49]

    The twelfth international conference on learning representations , year=

    On-policy distillation of language models: Learning from self-generated mistakes , author=. The twelfth international conference on learning representations , year=

  21. [50]

    International conference on machine learning , pages=

    Self-imitation learning , author=. International conference on machine learning , pages=. 2018 , organization=

    2018

  22. [54]

    Problems and Projects , url =

    Seven strictures on similarity , author =. Problems and Projects , url =

  23. [55]

    On-policy distillation of language models: Learning from self-generated mistakes

    Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. In The twelfth international conference on learning representations, 2024

    2024

  24. [56]

    Rl for reasoning by adaptively revealing rationales

    Mohammad Hossein Amani, Aryo Lotfi, Nicolas Mario Baldwin, Samy Bengio, Mehrdad Farajtabar, Emmanuel Abbe, and Robert West. Rl for reasoning by adaptively revealing rationales. arXiv preprint arXiv:2506.18110, 2025

    arXiv 2025

  25. [57]

    Hindsight experience replay

    Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017

    2017

  26. [58]

    Matharena: Evaluating llms on uncontaminated math competitions, February 2025

    Mislav Balunović, Jasper Dekoninck, Ivo Petrov, Nikola Jovanović, and Martin Vechev. Matharena: Evaluating llms on uncontaminated math competitions, February 2025. https://matharena.ai/

    2025

  27. [59]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021

    Pith/arXiv arXiv 2021

  28. [60]

    Self-evolving curriculum for llm reasoning

    Xiaoyin Chen, Jiarui Lu, Minsu Kim, Dinghuai Zhang, Jian Tang, Alexandre Pich \'e , Nicolas Gontier, Yoshua Bengio, and Ehsan Kamalloo. Self-evolving curriculum for llm reasoning. arXiv preprint arXiv:2505.14970, 2025

    arXiv 2025

  29. [61]

    Off-policy actor-critic

    Thomas Degris, Martha White, and Richard S Sutton. Off-policy actor-critic. arXiv preprint arXiv:1205.4839, 2012

    Pith/arXiv arXiv 2012

  30. [62]

    Agenther: Hindsight experience replay for llm agent trajectory relabeling

    Liang Ding. Agenther: Hindsight experience replay for llm agent trajectory relabeling. arXiv preprint arXiv:2603.21357, 2026

    Pith/arXiv arXiv 2026

  31. [63]

    Omni-math: A universal olympiad level mathematic benchmark for large language models

    Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, Zhengyang Tang, Benyou Wang, Daoguang Zan, Shanghaoran Quan, Ge Zhang, Lei Sha, Yichang Zhang, Xuancheng Ren, Tianyu Liu, and Baobao Chang. Omni-math: A universal olympiad level mathematic benchmark for large language models. arXiv preprint ar...

    Pith/arXiv arXiv 2024

  32. [64]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025 a

    Pith/arXiv arXiv 2025

  33. [65]

    Ggrpo-a: Guided group relative policy optimization with adaptive guidance

    Yongxin Guo, Wenbo Deng, Zhenglin Cheng, and Xiaoying Tang. Ggrpo-a: Guided group relative policy optimization with adaptive guidance. arXiv preprint arXiv:2508.13023, 2025 b

    Pith/arXiv arXiv 2025

  34. [66]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021

    2021

  35. [67]

    Lo RA : Low-rank adaptation of large language models

    Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lo RA : Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022

    2022

  36. [68]

    Sample-efficient online learning in lm agents via hindsight trajectory rewriting

    Michael Y Hu, Benjamin Van Durme, Jacob Andreas, and Harsh Jhamtani. Sample-efficient online learning in lm agents via hindsight trajectory rewriting. arXiv preprint arXiv:2510.10304, 2025

    arXiv 2025

  37. [69]

    Blending supervised and reinforcement fine-tuning with prefix sampling

    Zeyu Huang, Tianhao Cheng, Zihan Qiu, Zili Wang, Yinghui Xu, Edoardo M Ponti, and Ivan Titov. Blending supervised and reinforcement fine-tuning with prefix sampling. arXiv preprint arXiv:2507.01679, 2025

    Pith/arXiv arXiv 2025

  38. [70]

    u botter, Frederike L \

    Jonas H \"u botter, Frederike L \"u beck, Lejs Behric, Anton Baumann, Marco Bagatella, Daniel Marta, Ido Hakimi, Idan Shenfeld, Thomas Kleine Buening, Carlos Guestrin, et al. Reinforcement learning via self-distillation. arXiv preprint arXiv:2601.20802, 2026

    Pith/arXiv arXiv 2026

  39. [71]

    Openai o1 system card

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024

    Pith/arXiv arXiv 2024

  40. [72]

    Vcrl: Variance-based curriculum reinforcement learning for large language models

    Guochao Jiang, Wenfeng Feng, Guofeng Quan, Chuzhan Hao, Yuewei Zhang, Guohua Liu, and Hao Wang. Vcrl: Variance-based curriculum reinforcement learning for large language models. arXiv preprint arXiv:2509.19803, 2025

    arXiv 2025

  41. [73]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023

    2023

  42. [74]

    Adacurl: Adaptive curriculum reinforcement learning with invalid sample mitigation and historical revisiting

    Renda Li, Hailang Huang, Fei Wei, Feng Xiong, Yong Wang, and Xiangxiang Chu. Adacurl: Adaptive curriculum reinforcement learning with invalid sample mitigation and historical revisiting. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 23123--23131, 2026

    2026

  43. [75]

    Staying in the sweet spot: Responsive reasoning evolution via capability-adaptive hint scaffolding

    Ziheng Li, Zexu Sun, Jinman Zhao, Erxue Min, Yongcheng Zeng, Hui Wu, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Xu Chen, et al. Staying in the sweet spot: Responsive reasoning evolution via capability-adaptive hint scaffolding. arXiv preprint arXiv:2509.06923, 2025

    Pith/arXiv arXiv 2025

  44. [76]

    Off-policy temporal-difference learning with function approximation

    Doina Precup, Richard S Sutton, and Sanjoy Dasgupta. Off-policy temporal-difference learning with function approximation. In ICML, pages 417--424, 2001

    2001

  45. [77]

    Pope: Learning to reason on hard problems via privileged on-policy exploration

    Yuxiao Qu, Amrith Setlur, Virginia Smith, Ruslan Salakhutdinov, and Aviral Kumar. Pope: Learning to reason on hard problems via privileged on-policy exploration. arXiv preprint arXiv:2601.18779, 2026

    arXiv 2026

  46. [78]

    Qwen2.5: A party of foundation models, September 2024

    Qwen. Qwen2.5: A party of foundation models, September 2024. https://qwenlm.github.io/blog/qwen2.5/

    2024

  47. [79]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    Pith/arXiv arXiv 2017

  48. [80]

    Reuse your flops: Scaling rl on hard problems by conditioning on very off-policy prefixes

    Amrith Setlur, Zijian Wang, Andrew Cohen, Paria Rashidinejad, and Sang Michael Xie. Reuse your flops: Scaling rl on hard problems by conditioning on very off-policy prefixes. arXiv preprint arXiv:2601.18795, 2026

    arXiv 2026

  49. [81]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    Pith/arXiv arXiv 2024

  50. [82]

    Hybridflow: A flexible and efficient rlhf framework

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19256, 2024

    Pith/arXiv arXiv 2024

  51. [83]

    Beyond human data: Scaling self-training for problem-solving with language models

    Avi Singh, John D Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J Liu, James Harrison, Jaehoon Lee, Kelvin Xu, et al. Beyond human data: Scaling self-training for problem-solving with language models. arXiv preprint arXiv:2312.06585, 2023

    Pith/arXiv arXiv 2023

  52. [84]

    Trial and error: Exploration-based trajectory optimization of llm agents

    Yifan Song, Da Yin, Xiang Yue, Jie Huang, Sujian Li, and Bill Yuchen Lin. Trial and error: Exploration-based trajectory optimization of llm agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7584--7600, 2024

    2024

  53. [85]

    Gates: Self-distillation under privileged context with consensus gating

    Alex Stein, Furong Huang, and Tom Goldstein. Gates: Self-distillation under privileged context with consensus gating. arXiv preprint arXiv:2602.20574, 2026

    arXiv 2026

  54. [86]

    AIME problem set 1983--2024

    Hemish Veeraboina. AIME problem set 1983--2024. https://www.kaggle.com/datasets/hemishveeraboina/aime-problem-set-1983-2024, 2023

    1983

  55. [87]

    Dump: Automated distribution-level curriculum learning for rl-based llm post-training

    Zhenting Wang, Guofeng Cui, Yu-Jhe Li, Kun Wan, and Wentian Zhao. Dump: Automated distribution-level curriculum learning for rl-based llm post-training. arXiv preprint arXiv:2504.09710, 2025

    arXiv 2025

  56. [88]

    Learning to reason under off-policy guidance

    Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. Learning to reason under off-policy guidance. arXiv preprint arXiv:2504.14945, 2025

    Pith/arXiv arXiv 2025

  57. [89]

    Depth-breadth synergy in rlvr: Unlocking llm reasoning gains with adaptive exploration

    Zhicheng Yang, Zhijiang Guo, Yinya Huang, Yongxin Wang, Dongchun Xie, Hanhui Li, Yiwei Wang, Xiaodan Liang, and Jing Tang. Depth-breadth synergy in rlvr: Unlocking llm reasoning gains with adaptive exploration. arXiv preprint arXiv:2508.13755, 2025

    Pith/arXiv arXiv 2025

  58. [90]

    Dapo: An open-source llm reinforcement learning system at scale

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025

    Pith/arXiv arXiv 2025

  59. [91]

    Star: Bootstrapping reasoning with reasoning

    Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35: 0 15476--15488, 2022

    2022

  60. [92]

    Exgrpo: Learning to reason from experience

    Runzhe Zhan, Yafu Li, Zhi Wang, Xiaoye Qu, Dongrui Liu, Jing Shao, Derek F Wong, and Yu Cheng. Exgrpo: Learning to reason from experience. arXiv preprint arXiv:2510.02245, 2025

    arXiv 2025

  61. [93]

    Stephint: Multi-level stepwise hints enhance reinforcement learning to reason

    Kaiyi Zhang, Ang Lv, Jinpeng Li, Yongbo Wang, Feng Wang, Haoyuan Hu, and Rui Yan. Stephint: Multi-level stepwise hints enhance reinforcement learning to reason. arXiv preprint arXiv:2507.02841, 2025 a

    Pith/arXiv arXiv 2025

  62. [94]

    Clpo: Curriculum learning meets policy optimization for llm reasoning

    Shijie Zhang, Guohao Sun, Kevin Zhang, Xiang Guo, and Rujun Guo. Clpo: Curriculum learning meets policy optimization for llm reasoning. arXiv preprint arXiv:2509.25004, 2025 b

    Pith/arXiv arXiv 2025

  63. [95]

    On-policy rl meets off-policy experts: Harmonizing supervised fine-tuning and reinforcement learning via dynamic weighting

    Wenhao Zhang, Yuexiang Xie, Yuchang Sun, Yanxi Chen, Guoyin Wang, Yaliang Li, Bolin Ding, and Jingren Zhou. On-policy rl meets off-policy experts: Harmonizing supervised fine-tuning and reinforcement learning via dynamic weighting. arXiv preprint arXiv:2508.11408, 2025 c

    arXiv 2025

  64. [96]

    Evaluating the performance of large language models on gaokao benchmark

    Xiaotian Zhang, Chunyang Li, Yi Zong, Zhengyu Ying, Liang He, and Xipeng Qiu. Evaluating the performance of large language models on gaokao benchmark. arXiv preprint arXiv:2305.12474, 2023

    Pith/arXiv arXiv 2023

  65. [97]

    Scaf-grpo: Scaffolded group relative policy optimization for enhancing llm reasoning

    Xichen Zhang, Sitong Wu, Yinghao Zhu, Haoru Tan, Shaozuo Yu, Ziyi He, and Jiaya Jia. Scaf-grpo: Scaffolded group relative policy optimization for enhancing llm reasoning. arXiv preprint arXiv:2510.19807, 2025 d

    arXiv 2025

  66. [98]

    Bread: Branched rollouts from expert anchors bridge sft & rl for reasoning

    Xuechen Zhang, Zijian Huang, Yingcong Li, Chenshun Ni, Jiasi Chen, and Samet Oymak. Bread: Branched rollouts from expert anchors bridge sft & rl for reasoning. arXiv preprint arXiv:2506.17211, 2025 e

    Pith/arXiv arXiv 2025

  67. [99]

    Self-distilled reasoner: On-policy self-distillation for large language models

    Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. Self-distilled reasoner: On-policy self-distillation for large language models. arXiv preprint arXiv:2601.18734, 2026

    Pith/arXiv arXiv 2026