Pith. sign in

REVIEW 4 major objections 5 minor 4 cited by

Training LLMs with abstract hints, not partial answers, stabilizes RL reasoning and improves generalization.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 10:26 UTC pith:ZSFYLQOI

load-bearing objection HINT's two-stage heuristic-hint rescue is a genuinely new and plausible idea, but the paper's no-leakage story has an unaddressed off-policy bias and the single-run results don't yet support the strong claims. the 4 major comments →

arxiv 2510.09388 v2 pith:ZSFYLQOI submitted 2025-10-10 cs.LG cs.CL

Don't Tell the Answer, Truly Guide the Reasoning During RL Rollouts

Xinyi Wang , Jinyi Han , Zishang Jiang , Tingyun Li , Jiaqing Liang , Sihang Jiang , Zhaoqian Dai , Shuguang Ma
show 2 more authors
Fei Yu Yanghua Xiao
This is my paper
classification cs.LG cs.CL
keywords reinforcement learningLLM reasoningGRPOreward sparsityheuristic hintstraining affinitydistributional mismatchchain-of-thought
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Reinforcement learning for long-chain reasoning runs into a wall when a task is harder than the model: every sampled attempt is wrong, rewards go flat, and the standard GRPO update has nothing to learn from. Existing remedies inject outside data — either mixing in supervised examples or giving the model a chunk of the ground-truth answer — but the paper argues these create a distributional mismatch that inflates the training reward while leaving test accuracy stagnant. The paper's central move is to name and quantify this mismatch as low 'training affinity,' measured by how much of the policy update survives clipping and how consistent surviving updates are. It then proposes HINT, a two-stage rollout that, when all attempts fail, resamples from the same model conditioned on a teacher-generated heuristic hint (a conceptual nudge, not an answer), while the policy optimization prompt stays the original question so hints never become a crutch. The claim is that this preserves high affinity and turns failed rollouts into useful learning signals, yielding higher, more stable accuracy and transferable reasoning skills.

Core claim

The paper's core claim is that answer-level hints produce an illusion of high reward: training reward rises sharply while test accuracy stagnates, because the hint pulls the policy far from its current distribution, generating large importance ratios that get clipped away. HINT replaces those answer hints with heuristic 'meta-hints' — a single non-spoiler insight that tells the model how to think about the problem without revealing the solution. When all rollouts for a problem are wrong, the model resamples conditioned on the hinted query; the gradient is then computed from these hinted trajectories, but the policy prompt is the original question, decoupling exploration from optimization. Th

What carries the argument

The load-bearing object is the Affinity metric, defined as Affinity = EUR · exp(-UC/τ), where EUR is the weighted fraction of policy updates whose log-importance ratio stays inside the clipping trust region and UC is the weighted standard deviation of those ratios. It is meant to be a single, online monitor of exploration efficiency and training stability. The second mechanism is HINT's two-stage rollout: a standard GRPO stage, plus a rescue stage that activates only when all G rollouts are incorrect, in which the model resamples conditioned on qh (question + heuristic hint) while the policy objective still uses the original query q. The heuristic hints themselves are generated by a stronger

Load-bearing premise

The method treats trajectories sampled under a hint-augmented question as if they were on-policy data for the original question, relying on the decoupled policy prompt to prevent the model from learning a hint dependency — but no off-policy correction is applied to the gradient for that conditioning shift.

What would settle it

Take a fixed set of hard problems and run HINT with hints drawn randomly from a pool of unrelated heuristic sentences instead of problem-specific teacher hints. If the model still improves at the same rate and reaches the same accuracy, the improvement is driven by the extra sample budget created by resampling, not by the hint content — falsifying the claimed mechanism. Conversely, if answer-level hints on the same problems produce high Affinity while test accuracy stays flat, Affinity is not predicting generalization.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If HINT is right, reward curves alone are a deceptive success signal; monitoring Affinity (or its components) during RL training gives a more faithful picture of whether the policy is actually learning.
  • Failed rollouts need not be wasted: conditioning a resample on a heuristic hint converts an all-incorrect group into a usable gradient without the distributional damage of answer leakage.
  • Training on heuristic hints transfers: the model reapplies the hinted reasoning pattern (e.g., proof by contradiction) to unseen problems, so gains show up on out-of-distribution benchmarks, not just the training distribution.
  • The advantage of heuristic over answer-level hints scales with model capacity: larger backbones extract successful trajectories from heuristic hints more often, so HINT's gains widen as model size grows.
  • HINT's higher entropy means the policy keeps exploring instead of collapsing to a deterministic path, which is a concrete, measurable consequence beyond final accuracy.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the paper's 'decoupled prompt' trick is a form of implicit off-policy correction; if the hinted query distribution differs strongly from the original, the gradient is still an off-distribution update. A direct test is to measure the KL divergence between rollout distributions under qh and q and check whether Affinity drops when that divergence is large.
  • Editorial inference: Affinity could be used as a practical early-stopping or data-filtering signal — e.g., pause hinting when Affinity saturates, since the paper's curves show HINT converging toward GRPO-like affinity in later training.
  • Editorial inference: the method should transfer to any domain with verifiable rewards and a teacher capable of generating non-spoiler insights (code correctness, theorem proving), not just mathematics.
  • Editorial inference: a natural scaling experiment is to vary hint abstractness (from random generic sentences to answer-level) and measure the Affinity/accuracy frontier; the paper's comparison suggests a sweet spot between too vague and too revealing.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper identifies what it calls low training affinity in RL for LLM reasoning: when external guidance (answer-level hints or off-policy SFT data) is used to rescue sparse-reward rollouts, it causes frequent clipping and unstable importance ratios. It proposes an Affinity metric, defined as EUR times exp(-UC/τ), to monitor this phenomenon, and a framework called HINT that, when all rollouts on a problem are incorrect, resamples trajectories conditioned on a teacher-generated heuristic hint, then updates with a GRPO-style objective. Experiments on Qwen2.5-7B and Qwen2.5-3B across seven benchmarks report gains over GRPO and several hint-based baselines, together with training-dynamics plots of EUR, UC, Affinity, and entropy. The paper includes code, detailed experimental settings, and an appendix with prompts and algorithms.

Significance. If the claimed mechanism held, the paper would make a useful contribution: a diagnostic for off-policy guidance in reasoning RL and a practical hinting procedure that may avoid the reward-inflation pathology of answer-level hints. The experimental breadth is a strength, as is the decision to release code and report training dynamics. However, the central no-leakage claim is not supported by the formal objective as written: the update rule in Eq. (8) optimizes under the hinted prompt, not the original inference-time prompt. In addition, the main accuracy comparison in Table 1 is reported without error bars or multiple seeds, and several benchmark cells are not wins for HINT. The Affinity metric is also partly definitional, since it is built from the two quantities HINT is designed to control. These issues do not necessarily invalidate the empirical approach, but they make the paper's strongest claims premature.

major comments (4)
  1. [§2.3, Eq. (7)–(8), Algorithm 1] The claimed decoupling of the rollout prompt from the policy prompt is contradicted by the gradient definition. Eq. (8) sets q* = qh whenever all first-stage rollouts are incorrect, and Eq. (7) computes the importance ratio r_i,t(θ) as πθ(o_i,t | q*, o_i,<t) / πθold(o_i,t | q*, o_i,<t). Thus every hint-rescued update explicitly increases the likelihood of trajectories under the hint-augmented query qh, not under the evaluation-time question q. Algorithm 1 lines 13–19 implement exactly this: trajectories are resampled under qh and then used to maximize the GRPO objective. No importance weight or off-policy correction accounts for the qh→q conditioning shift, so the clipped surrogate is not an unbiased estimator of the inference-time expected reward. The Affinity curves in Fig. 4, computed on these same updates, therefore show stable learning of the hinted objective, not transfer to q. The
  2. [Table 1] The abstract and §3.2 claim that HINT 'consistently outperforms strong baselines,' but Table 1 reports a single run per method with no error bars, no multiple seeds, and no significance tests. Several cells are not HINT wins: Minerva 7B is 31.0 for HINT versus 31.2 for GRPO; GPQA 3B is 11.8 for HINT versus 12.0 for QuestA and 11.8 for GRPO/BREAD; AIME 7B is 13.3 versus 13.2 for CHORD. The average gaps are also small (7B: 59.7 vs 58.2 for GRPO; 3B: 29.9 vs 28.3 for GRPO). At minimum, report results over at least three seeds with standard deviations, and preferably paired significance tests, before claiming consistent superiority. This is load-bearing for the paper's main empirical contribution.
  3. [§2.2, Eq. (1)–(6)] The Affinity metric is constructed directly from EUR and UC, which are themselves the two quantities HINT's design is intended to control: clipping frequency and importance-ratio variance. Calling HINT's training higher-Affinity is therefore partly definitional. The independent validation must come from held-out accuracy, but, as noted above, Table 1 lacks statistical power. In addition, the threshold δ in Eqs. (1)–(6) is never specified in the hyperparameters; Appendix A lists the clipping parameter ε = 0.2 but not δ, and τ is fixed as δ/2 with no sensitivity analysis. State how δ is chosen; if it is intended to equal the clipping threshold, say so explicitly.
  4. [§3.1, Appendix B.5] Hints are generated by Qwen2.5-72B-Instruct with a prompt that instructs the teacher not to reveal numbers or the final answer, but no verification is reported that the generated hints actually comply. Since the entire distinction from answer-level baselines rests on hints not disclosing solutions, the manuscript should include a small leakage audit (e.g., 100 sampled hints judged for answer content or whether a third-party model can solve the problem from the hint alone). Without this, the mechanism claim remains unverified.
minor comments (5)
  1. [Eq. (7) vs Appendix A.1] The objective in Eq. (7) includes a KL penalty term -β D_KL(πθ || πref), but Appendix A.1 states kl_coef is set to 0.0. Reconcile this inconsistency or remove the KL term from the displayed objective.
  2. [Table 1 and §3.2] The header 'Olympaid' is a typo for 'OlympiadBench'; §3.3 contains 'avoid casing large distributional shifts' (should be 'causing'); the abstract has a missing space in 'we introduceAffinity'.
  3. [Figure 1] The example is confusing: for the sum 1..1000 the displayed 'answer' is 1001, but the correct sum is 500500; presumably 1001 is the first-plus-last pair. This should be corrected or relabeled so the contrast between answer-level and heuristic hints is clear.
  4. [§2.2, Eqs. (2)–(5)] The notation for EUR/UC uses a generic sample index i with actions a_i and states s_i, while Eq. (7) uses token-level o_i,t. Clarify whether EUR/UC are computed at the token level, trajectory level, or both, since the weighting in Eq. (2) depends on this choice.
  5. [§2.3, Eq. (7)] The expectation in Eq. (7) is written over x ∼ D_e, but the distribution D_e is not defined; presumably it is the problem distribution. Please define all notation in one place.

Circularity Check

0 steps flagged

No significant circularity: HINT's central claims are supported by external benchmark evaluations, and the Affinity metric is a diagnostic rather than a fitted input or derivation target.

full rationale

The paper's main empirical claims are not derived from its definitions. HINT is evaluated on held-out test sets (Table 1) and through independent behavioral analyses (Figures 2, 5, 7), all of which are external to the Affinity/EUR/UC construction. Equations (1)-(6) define monitoring statistics from log-importance ratios and clipping behavior; these are not fitted parameters used to produce the benchmark results, and the claim that HINT 'achieves higher Affinity' is an empirical observation, not a definitional consequence. Even if EUR and UC partly repackage standard PPO clipping diagnostics under the name Affinity, that is a framing choice rather than a circular derivation, and the accuracy comparisons do not depend on the metric. The one substantive concern is the gap between Eqs. (7)-(8), where q* switches to the hint-augmented qh for sparse rollouts, and the text's assurance that the policy prompt is 'restricted to the original problem only.' This is a possible off-policy / no-leakage flaw in the method's justification and would be a correctness risk, but it is not a circular reduction: the evaluation data are not constructed from the training objective. There is also no load-bearing self-citation chain, no imported uniqueness theorem, and no ansatz smuggled in via citation. Under the rule that circularity requires exhibiting a specific definitional or fitted equivalence, no such step is present.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 1 invented entities

The paper's central claim rests on two unvalidated assumptions: (1) that the Affinity metric's functional form captures training quality, and (2) that hint-conditioned rollouts can be used to optimize a policy that will later be evaluated without hints. Both are domain assumptions rather than derived results. Only standard trust-region RL math is imported without question.

free parameters (2)
  • τ (Affinity temperature) = δ/2
    Eq 6 sets τ=δ/2 with no derivation or ablation; the Affinity score's magnitude and ranking depend on this hand-chosen value.
  • δ (EUR/UC trust-region threshold) = not specified in main text; assumed equal to clipping ϵ=0.2
    Eqs 1 and 4 use δ without defining its value; if chosen after observing training curves, the metric is post hoc.
axioms (4)
  • domain assumption A policy update that stays within a clipped trust region and has low variance of log-importance ratios is a reliable proxy for training quality and generalization.
    Section 2.2 defines Affinity from EUR and UC and uses it to explain generalization differences; this equivalence is assumed, not derived.
  • domain assumption Heuristic hints generated by Qwen2.5-72B-Instruct from the question and final answer do not reveal the answer or the specific numbers used in the solution.
    Section B.5 prompt constraints assert this, but the paper does not independently verify that the generated hints are always spoiler-free.
  • domain assumption The importance ratio is computed using q* (original or hinted prompt) as the conditioning context, and no off-policy correction is needed when switching from q to qh.
    Eqs 7-8 and Algorithm 1 replace the rollout query with qh when rewards are sparse; the update is assumed to transfer to q-only inference because the policy prompt is decoupled.
  • standard math Standard GRPO/PPO trust-region clipping machinery is valid for policy optimization.
    The paper relies on Schulman et al. 2015/2017 as accepted background; this is not the target result.
invented entities (1)
  • Affinity metric (EUR·exp(−UC/τ)) no independent evidence
    purpose: A diagnostic intended to measure compatibility between external guidance and the model's current policy during RL training.
    Affinity is defined entirely from importance-ratio clipping statistics — the same quantities HINT's design directly controls. The paper provides no external validation that Affinity predicts generalization independently of the training curves used to construct it.

pith-pipeline@v1.3.0-alltime-deepseek · 14141 in / 9669 out tokens · 97663 ms · 2026-08-04T10:26:30.581695+00:00 · methodology

0 comments
read the original abstract

Reinforcement Learning (RL) has become a key driver for enhancing the long chain-of-thought (CoT) reasoning capabilities of Large Language Models (LLMs). However, prevalent methods like GRPO often fail when task difficulty exceeds model capacity, leading to reward sparsity and inefficient training. Prior work attempts to mitigate this with off-policy data, but such methods often induce severe distributional mismatches that destabilize policy updates. In this work, we identify a core issue underlying these failures, which we term low training affinity, and introduce Affinity, the first quantitative metric for monitoring the compatibility between external guidance and the model's intrinsic policy. To address this, we propose HINT, an adaptive framework designed to enhance reasoning capabilities while explicitly preserving high Affinity. First, instead of revealing partial answers, HINT supplies Meta-Hints, which act as abstract cognitive scaffolding to guide the model in articulating solutions independently. Second, to ensure stability, we integrate Affinity-Aware Policy Optimization (AAPO), which dynamically modulates the learning objective based on the Affinity. Extensive experiments across diverse benchmarks demonstrate that HINT consistently outperforms strong baselines, while exhibiting superior stability and robust generalization to out-of-distribution tasks. Code is available at https://github.com/ViviqwerAsd/HINT.

Figures

Figures reproduced from arXiv: 2510.09388 by Fei Yu, Jiaqing Liang, Jinyi Han, Shuguang Ma, Sihang Jiang, Tingyun Li, Xinyi Wang, Yanghua Xiao, Zhaoqian Dai, Zishang Jiang.

Figure 1
Figure 1. Figure 1: Comparison of Hint Mechanisms and Their Impact on Learning. The answer-level hint provides an explicit [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: A comparison of training rewards (top) and [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The HINT Framework: An Adaptive Two-Stage Rollout Process. HINT operates in two stages. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: We record the EUR, UC, and Affinity metrics across different training processes to investigate the impact of various off-policy strategies on training. Left: EUR during training; Middle: UC during training; Right: Affinity during training. Overall, HINT most effectively alleviates the EUR collapse, avoids persistently high UC, and achieves higher Affinity, thereby enabling more stable and efficient trainin… view at source ↗
Figure 5
Figure 5. Figure 5: Sampling Efficiency of HINT and GRPO at Different Training Stages. Under an equal budget, HINT yields 1,485 more valid samples (top) and achieves a 18.9% higher final proportion of valid samples (bottom). As illustrated in [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: HINT Prevents Entropy Collapse and En￾courages Sustained Exploration. HINT maintains a high entropy level, especially in the early stages, and sta￾bilizes at a significantly higher value. This demonstrates that HINT’s heuristic guidance fosters more continuous and diverse exploration, preventing premature policy convergence. final answer in a model’s generated trajectory. We use the Math-Verify tool for au… view at source ↗
Figure 7
Figure 7. Figure 7: Accuracy of Different Methods. HINT Achieves Higher Final Accuracy Despite Slower Ini￾tial Convergence. B.2 Details of HINT’s Accuracy Our results reveal an interesting trade-off: while the off-policy guidance from HINT may initially slow the rate of convergence, it ultimately enables the model to achieve a higher performance ceiling. As shown in [PITH_FULL_IMAGE:figures/full_fig_p013_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. It Takes 8 Tokens: Weak-to-Strong Off-Policy RL via Auxiliary Branches

    cs.AI 2026-05 conditional novelty 6.0

    W2SPO trains a reasoning LLM by inserting short 8-token branches proposed by a weaker model into its own rollouts and updating only those branch tokens, improving Pass@1 over vanilla GRPO (64.2% vs 62.3%) with a 3.55x...

  2. No More Stale Feedback: Co-Evolving Critics for Open-World Agent Learning

    cs.AI 2026-01 unverdicted novelty 6.0

    ECHO jointly optimizes policy and critic via co-evolution, cascaded rollouts, and saturation-aware shaping to deliver non-stale feedback and higher success in open-world LLM agent RL.

  3. LANG: Reinforcement Learning for Multilingual Reasoning with Language-Adaptive Hint Guidance

    cs.CL 2026-05 unverdicted novelty 5.0

    LANG combines language-adaptive hint guidance, progressive decay, and difficulty-tailored learning horizons in RL to boost non-English reasoning performance while preserving language consistency.

  4. N-GRPO: Embedding-Level Neighbor Mixing for Enhanced Policy Optimization

    cs.LG 2026-06 unverdicted novelty 3.0

    N-GRPO enhances GRPO via Semantic Neighbor Mixing of token embeddings to improve diversity and consistency in LLM math reasoning rollouts.

Reference graph

Works this paper leans on

40 extracted references · 27 linked inside Pith · cited by 4 Pith papers

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Arash Ahmadian, Chris Cremer, Matthias Gall \'e , Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet \"U st \"u n, and Sara Hooker. 2024. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint arXiv:2402.14740

    Pith/arXiv arXiv 2024

  4. [4]

    Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang, and Furu Wei. 2025. Reasoning with exploration: An entropy perspective. arXiv preprint arXiv:2506.14758

    Pith/arXiv arXiv 2025

  5. [5]

    Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. 2025. Sft memorizes, rl generalizes: A comparative study of foundation model post-training. arXiv preprint arXiv:2501.17161

    Pith/arXiv arXiv 2025

  6. [6]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457

    Pith/arXiv arXiv 2018

  7. [7]

    Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, and 1 others. 2025. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01456

    Pith/arXiv arXiv 2025

  8. [8]

    Wei Fu, Jiaxuan Gao, Xujie Shen, Chen Zhu, Zhiyu Mei, Chuyi He, Shusheng Xu, Guo Wei, Jun Mei, Jiashu Wang, and 1 others. 2025 a . Areal: A large-scale asynchronous reinforcement learning system for language reasoning. arXiv preprint arXiv:2505.24298

    Pith/arXiv arXiv 2025

  9. [9]

    Yuqian Fu, Tinghong Chen, Jiajun Chai, Xihuai Wang, Songjun Tu, Guojun Yin, Wei Lin, Qichao Zhang, Yuanheng Zhu, and Dongbin Zhao. 2025 b . Srft: A single-stage method with supervised and reinforcement fine-tuning for reasoning. arXiv preprint arXiv:2506.19767

    Pith/arXiv arXiv 2025

  10. [10]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948

    Pith/arXiv arXiv 2025

  11. [11]

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, and 1 others. 2024. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402.14008

    Pith/arXiv arXiv 2024

  12. [12]

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874

    Pith/arXiv arXiv 2021

  13. [13]

    Jian Hu. 2025. Reinforce++: A simple and efficient approach for aligning large language models. arXiv preprint arXiv:2501.03262

    Pith/arXiv arXiv 2025

  14. [14]

    Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung-Yeung Shum. 2025. Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model. arXiv preprint arXiv:2503.24290

    Pith/arXiv arXiv 2025

  15. [15]

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, and 1 others. 2024. Openai o1 system card. arXiv preprint arXiv:2412.16720

    Pith/arXiv arXiv 2024

  16. [16]

    Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. 2024. A survey on large language models for code generation. arXiv preprint arXiv:2406.00515

    Pith/arXiv arXiv 2024

  17. [17]

    Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, and 1 others. 2022. Solving quantitative reasoning problems with language models. Advances in neural information processing systems, 35:3843--3857

    2022

  18. [18]

    Jiazheng Li, Hong Lu, Kaiyue Wen, Zaiwen Yang, Jiaxuan Gao, Hongzhou Lin, Yi Wu, and Jingzhao Zhang. 2025. Questa: Expanding reasoning capacity in llms via question augmentation. arXiv preprint arXiv:2507.13266

    arXiv 2025

  19. [19]

    Zhihang Lin, Mingbao Lin, Yuan Xie, and Rongrong Ji. 2025. Cppo: Accelerating the training of group relative policy optimization-based reasoning models. arXiv preprint arXiv:2503.22342

    arXiv 2025

  20. [20]

    Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. 2025 a . Understanding r1-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783

    Pith/arXiv arXiv 2025

  21. [21]

    Ziru Liu, Cheng Gong, Xinyu Fu, Yaofang Liu, Ran Chen, Shoubo Hu, Suiyun Zhang, Rui Liu, Qingfu Zhang, and Dandan Tu. 2025 b . Ghpo: Adaptive guidance for stable and efficient llm reinforcement learning. arXiv preprint arXiv:2507.10628

    Pith/arXiv arXiv 2025

  22. [22]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. 2024. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling

    2024

  23. [23]

    John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. 2015. Trust region policy optimization. In International conference on machine learning, pages 1889--1897. PMLR

    2015

  24. [24]

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347

    Pith/arXiv arXiv 2017

  25. [25]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, and 1 others. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300

    Pith/arXiv arXiv 2024

  26. [26]

    Charlie Victor Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2025. Scaling llm test-time compute optimally can be more effective than scaling parameters for reasoning. In The Thirteenth International Conference on Learning Representations

    2025

  27. [27]

    Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, and 1 others. 2025. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599

    Pith/arXiv arXiv 2025

  28. [28]

    Qwen Team. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671

    Pith/arXiv arXiv 2024

  29. [29]

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, and 1 others. 2024. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. Advances in Neural Information Processing Systems, 37:95266--95290

    2024

  30. [30]

    Yuhui Wang, Hao He, and Xiaoyang Tan. 2020. Truly proximal policy optimization. In Uncertainty in artificial intelligence, pages 113--122. PMLR

    2020

  31. [31]

    Zhenting Wang, Guofeng Cui, Yu-Jhe Li, Kun Wan, and Wentian Zhao. 2025. Dump: Automated distribution-level curriculum learning for rl-based llm post-training. arXiv preprint arXiv:2504.09710

    arXiv 2025

  32. [32]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

    2022

  33. [33]

    Yixuan Even Xu, Yash Savani, Fei Fang, and Zico Kolter. 2025. Not all rollouts are useful: Down-sampling rollouts in llm reinforcement learning. arXiv preprint arXiv:2504.13818

    Pith/arXiv arXiv 2025

  34. [34]

    Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. 2025. Learning to reason under off-policy guidance. arXiv preprint arXiv:2504.14945

    Pith/arXiv arXiv 2025

  35. [35]

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, and 1 others. 2025. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476

    Pith/arXiv arXiv 2025

  36. [36]

    Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. 2025. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprint arXiv:2504.13837

    Pith/arXiv arXiv 2025

  37. [37]

    Wenhao Zhang, Yuexiang Xie, Yuchang Sun, Yanxi Chen, Guoyin Wang, Yaliang Li, Bolin Ding, and Jingren Zhou. 2025 a . On-policy rl meets off-policy experts: Harmonizing supervised fine-tuning and reinforcement learning via dynamic weighting. arXiv preprint arXiv:2508.11408

    arXiv 2025

  38. [38]

    Xuechen Zhang, Zijian Huang, Yingcong Li, Chenshun Ni, Jiasi Chen, and Samet Oymak. 2025 b . Bread: Branched rollouts from expert anchors bridge sft & rl for reasoning. arXiv preprint arXiv:2506.17211

    Pith/arXiv arXiv 2025

  39. [39]

    Rosie Zhao, Alexandru Meterez, Sham Kakade, Cengiz Pehlevan, Samy Jelassi, and Eran Malach. 2025. Echo chamber: Rl post-training amplifies behaviors learned in pretraining. arXiv preprint arXiv:2504.07912

    Pith/arXiv arXiv 2025

  40. [40]

    Tianyu Zheng, Tianshun Xing, Qingshui Gu, Taoran Liang, Xingwei Qu, Xin Zhou, Yizhi Li, Zhoufutu Wen, Chenghua Lin, Wenhao Huang, and 1 others. 2025. First return, entropy-eliciting explore. arXiv preprint arXiv:2507.07017

    Pith/arXiv arXiv 2025