Pith. sign in

REVIEW 2 major objections 7 minor 21 references

Closed-loop prefix control doubles GRPO accuracy on hard math

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

2026-07-09 02:43 UTC pith:NCDLRKUI

load-bearing objection Solid paper. The closed-loop prefix controller for GRPO is a genuinely new mechanism with strong main results. The attribution claim (controller vs. annealing) rests on single-seed ablations and needs tightening, but the core improvement over vanilla GRPO is well-established. the 2 major comments →

arxiv 2607.07674 v1 pith:NCDLRKUI submitted 2026-07-08 cs.LG cs.CL

Max Out GRPO Signal: Adaptive Trace Prefix Control for Hard Reasoning Problems

Vladislav Beliaev This is my paper
classification cs.LG cs.CL
keywords GRPOreinforcement learningprefix conditioningclosed-loop controldifficulty curriculummathematical reasoninglanguage model trainingdead zone
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Group Relative Policy Optimization (GRPO), a standard method for training language models to reason, has a structural blind spot: on problems the model almost never solves, all sampled attempts fail identically, producing zero gradient signal. The hardest problems, which matter most for learning, contribute nothing. This paper introduces AdaPrefix-GRPO, which treats the length of a correct solution prefix given to the model as a feedback-controlled dial on problem difficulty. Rather than fixing the prefix length once at the start, the method continuously adjusts it throughout training to hold each problem's success rate near 50 percent, the regime where GRPO's learning signal is strongest. It then anneals all prefixes to zero so the deployed model solves problems unaided. The central claim is that closed-loop control of prefix length, not prefixing itself, drives the improvement: fixed-prefix and global-decay baselines trail by roughly 12 and 11 percentage points respectively on held-out hard math problems. At matched training compute, AdaPrefix-GRPO improves pass@1 accuracy over vanilla GRPO by 22.2 points on a 0.6B model and 18.1 points on a 1.7B model, while roughly halving output trace length. The gain is larger for smaller models, which sit deeper in the dead zone. The method requires only a loss mask on prefix tokens and data preparation changes; the underlying GRPO trainer is otherwise unmodified.

Core claim

The paper identifies that the binding constraint on GRPO for hard problems is not model capacity but sampling failure: when no rollout in a group succeeds, the group-relative advantage is exactly zero and no gradient flows. Prefix length converts this binary barrier into a continuous, controllable dial on difficulty. The key finding is that the dial must be closed-loop rather than fixed, because the relationship between prefix length and success rate shifts as the policy improves, causing any static choice to drift out of the signal-maximizing regime. A simple root-finding controller (secant with bisection fallback) tracking a 50 percent success-rate set-point, combined with per-sample难度-d难度

What carries the argument

The central mechanism is a one-dimensional root-finding controller that adjusts a single global base prefix ratio to hold the batch-mean success rate k/G near a target (default 0.5). Per-sample difficulty offsets (static, estimated from a cold-start probe) compress the spread of success rates across problems of varying difficulty. An annealing envelope forces all prefixes to zero in the final training phase so the deployed policy needs no prefix. Gradients are masked on prefix tokens; the reward is computed on the full completion.

Load-bearing premise

The compute-matched comparison excludes the one-time calibration sweep and difficulty probe as amortized preprocessing. If a practitioner trains a model once rather than running multiple seeds or sweeps, this overhead is not amortized and the effective compute advantage over baselines shrinks.

What would settle it

Run AdaPrefix-GRPO with the calibration and difficulty-probe costs included in the training budget. If the gains over vanilla GRPO and fixed-prefix baselines shrink substantially under full-cost accounting, the claim that improvements are not attributable to extra compute weakens.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Smaller models stuck in GRPO's dead zone can be trained to surpass larger models trained with vanilla GRPO, shifting the cost-benefit tradeoff between scaling parameters and improving the training recipe.
  • The principle of closed-loop difficulty targeting via controllable scaffolds could extend beyond prefix injection to any mechanism that modulates problem difficulty, such as hint granularity, tool access, or context window budget.
  • The finding that traces shorten as a side effect suggests that holding problems at moderate success rates may preferentially reinforce efficient solution paths rather than lengthy exploration, a hypothesis with implications for inference cost at deployment.
  • If the 50 percent success-rate set-point generalizes beyond math reasoning, curriculum design for code generation or logical reasoning tasks could adopt the same closed-loop targeting rather than static difficulty filtering.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The calibration sweep and difficulty probe, while excluded from the training budget, represent non-trivial one-time inference compute. For practitioners training a single model on a single dataset without multiple seeds or sweeps, this overhead does not amortize and the effective compute advantage over vanilla GRPO shrinks.
  • The per-sample difficulty offset is a static heuristic that cannot correct individual mis-estimations during training. A genuine per-sample second control loop, if it could be made feasible with denser rollout estimates, might recover the remaining spread the global loop leaves unaddressed.
  • The method's reliance on reference solutions and binary verifiers means its applicability to domains with noisy rewards, multiple valid solution styles, or no available reference traces remains untested. The controller's behavior under reward noise may differ from the clean math setting.
  • If the 50 percent set-point is not universal but depends on reward noise and solution-path multiplicity (as the paper acknowledges), an adaptive target that itself responds to measured signal quality could outperform the fixed 0.5 default.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 7 minor

Summary. The paper introduces AdaPrefix-GRPO, a closed-loop controller that adjusts prefix length during GRPO training to hold the batch success rate near 50%, where group-relative gradient signal is maximized. The method addresses the 'dead zone' problem where all rollouts in a group fail and the gradient vanishes. Prefixes are annealed to zero so the deployed model operates without assistance. Experiments on hard math (DeepMath) across multiple model sizes and families show substantial gains over vanilla GRPO and fixed-prefix baselines at matched training FLOPs, with shorter traces and no pass@16 collapse. The experimental design is thorough: FLOPs-matched comparisons, multiple baselines isolating confounds, 3-seed main comparisons, cross-family validation, and component-level ablations.

Significance. The core contribution—treating prefix length as a tracked set-point rather than a fixed hyperparameter—is well-motivated and the mechanism is clearly articulated. The paper ships falsifiable predictions (the target sweep in Table 3 confirms the intermediate-optimum prediction), a parameter-light controller (secant/bisection with EMA, no learned components), and an honest limitations section. The FLOPs-matched comparison with a G=16 vanilla GRPO control and an iteration-matched view (Figure 3) are commendable controls that address the most obvious alternative explanations. The cross-family transfer (Qwen3.5, Gemma 4) rules out architecture-specific effects. The finding that a 0.6B model with AdaPrefix surpasses a 3x larger model with vanilla GRPO is practically significant.

major comments (2)
  1. Section 5.3, Table 3 (Control granularity row): The central attribution claim—that 'closed-loop control, not prefixing per se, drives the improvement'—rests on the comparison between base-only closed-loop (45.3) and PrefixRL+anneal (38.7), a gap of 6.6 pp. However, both of these are single-seed results with CI approximately +/-3.1, so the gap is roughly 2 sigma and could shrink with multiple seeds. The 3-seed main table comparison (AdaPrefix 48.2 vs PrefixRL 35.9, gap 12.3 pp) is confounded because AdaPrefix includes annealing while PrefixRL does not. The paper should either (a) add at least one more seed for the base-only vs PrefixRL+anneal comparison to tighten the attribution, or (b) explicitly acknowledge in the main text (not just the limitations) that the specific attribution to closed-loop control over the joint controller+annealing combination rests on single-seed evidence, and软化
  2. Section 4.2 (Compute accounting): The FLOPs-matched comparison excludes the one-time calibration sweep (4 rollouts x 512 problems x 5 grid points) and difficulty probe (4 rollouts x ~6.8K problems) as 'amortized preprocessing.' This is defensible when multiple runs share the same model and data, but the paper's claim that gains are 'not attributable to extra training compute' is stated without qualification in the abstract and main results. For a practitioner training a single model once, this overhead is not amortized and represents non-trivial inference compute. The paper should state the absolute cost of calibration+probe in FLOPs or wall-clock time and note the breakeven point (how many training runs are needed before amortization holds), so readers can assess the practical accessibility claim.
minor comments (7)
  1. The abstract states 'more than doubles GRPO's accuracy' for the 0.6B model (2.1x). Table 2 shows 19.6 -> 41.8, which is indeed 2.1x. However, the abstract also says '1.6x on Qwen3-1.7B' — Table 2 shows 30.1 -> 48.2, which is 1.6x. These are consistent, but the abstract does not mention that these are DeepMath held-out numbers specifically; adding 'on held-out hard math' would avoid overpromising.
  2. Table 1: PrefixRL shows a slight regression on GSM8K (90.4 vs 91.3 for vanilla GRPO). The text attributes this to 'over-prefixing easy problems.' It would help to verify whether the per-sample offset mechanism in AdaPrefix (which sets rho=0 on easy problems) is specifically responsible for the GSM8K recovery (92.4), or whether the controller alone suffices. A one-line mention of whether the GSM8K slice was tracked during training would clarify this.
  3. Section 5.4 ('Why traces shorten'): The hypothesis that holding k/G near 0.5 'rewards efficient correct completions rather than long, lucky explorations' is plausible but unverified. The paper itself notes 'We treat this as a hypothesis to verify with length-conditioned analyses at rho=0.' This is fine as stated, but the claim in the abstract and Table 1 that traces are 'roughly halved' is a headline result; a brief note in the results section (not just the analysis) flagging that the mechanism is hypothesized would be more transparent.
  4. Figure 2a is labeled as 'illustrative' in the caption. Since the paper has actual calibration data (512 problems x 5 grid points), replacing the illustrative panel with a real measured kappa(rho) curve would strengthen the empirical foundation at no additional cost.
  5. The paper does not release training code or per-step data (acknowledged in Limitations). Given that the method is described as 'data preparation plus a loss mask on prefix tokens; the trainer is otherwise stock,' releasing the data preparation script would be low-effort and would significantly aid reproducibility.
  6. Section 4.2: The offset map delta is described as 'a small monotone (linear) map with a hand-tuned slope, spanning +/-0.15 around the base across the difficulty range.' The exact slope and how the difficulty range is normalized (quantile? raw pass rate?) are not specified. A one-line formula or a footnote with the exact mapping would be useful for reproduction.
  7. References to concurrent work (Setlur et al. 2026, Qu et al. 2026) use future-dated arXiv IDs. If these are genuinely concurrent submissions, noting 'concurrent work' explicitly in the Related Work section (as is done) is sufficient, but ensuring the arXiv IDs are correct at publication time would help readers.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the careful reading and the constructive recommendation. Both major comments are well-taken and will be addressed in revision. Below we respond point by point.

read point-by-point responses

  1. Referee: Major Comment 1 (Section 5.3, Table 3): The attribution claim that 'closed-loop control, not prefixing per se, drives the improvement' rests on single-seed evidence (base-only closed-loop 45.3 vs PrefixRL+anneal 38.7, gap 6.6 pp, CI ±3.1, roughly 2 sigma). The 3-seed main table comparison (AdaPrefix 48.2 vs PrefixRL 35.9) is confounded because AdaPrefix includes annealing while PrefixRL does not. The referee requests either (a) additional seeds for the base-only vs PrefixRL+anneal comparison, or (b) explicit acknowledgment in the main text that the specific attribution rests on single-seed evidence.

    Authors: The referee is correct on both counts. The 3-seed main-table comparison (48.2 vs 35.9) does conflate closed-loop control with annealing, since AdaPrefix-GRPO includes the annealing envelope and PrefixRL does not. The clean attribution to closed-loop control specifically—rather than to the joint controller+annealing combination—rests on the single-seed Table 3 rows: base-only closed-loop (45.3) vs PrefixRL+anneal (38.7), a gap of 6.6 pp with CI ±3.1, i.e., approximately 2 sigma. This is not the level of evidence we would want for a central attribution claim, and we agree it should be characterized honestly in the main text rather than only in the limitations section. We will revise Section 5.3 to state explicitly that the specific attribution to closed-loop control over the joint controller+annealing combination rests on single-seed evidence and that the gap, while directionally consistent with the claim, is not conclusive at conventional significance thresholds. We will also soften the related claim in Section 2 ('We additionally show it is closed-loop control, not prefixing per se, that carries the gain') to reflect this qualification. Regarding option (a): we will attempt to add at least one additional seed for the base-only closed-loop and PrefixRL+anneal rows during the revision period. If compute permits, this would tighten the CI; if not, the main-text qualification will stand. Either way, the revision will not overstate the strength of the attribution. revision: partial

  2. Referee: Major Comment 2 (Section 4.2, Compute accounting): The FLOPs-matched comparison excludes the calibration sweep (4 rollouts × 512 problems × 5 grid points) and difficulty probe (4 rollouts × ~6.8K problems) as 'amortized preprocessing.' The abstract and main results state gains are 'not attributable to extra training compute' without qualification. For a practitioner training a single model once, this overhead is not amortized and represents non-trivial inference compute. The referee requests the absolute cost of calibration+probe in FLOPs or wall-clock time, and the breakeven point for amortization.

    Authors: The referee is right that the unqualified statement in the abstract and main results is misleading for a practitioner training a single model once. The calibration sweep and difficulty probe are real inference compute that we exclude from the training budget only because they are shared across all runs on the same model and data (seeds, ablations, hyperparameter sweeps). For a single-run practitioner this overhead is not amortized and should be disclosed. We will add the following to Section 4.2: (1) The absolute cost of calibration+probe in terms of total rollout count: the calibration sweep performs 4 × 512 × 5 = 10,240 rollouts and the difficulty probe performs 4 × ~6,800 = ~27,200 rollouts, for a combined ~37,440 short rollouts. (2) Expressed as a fraction of the training budget: for the 1.7B model, this preprocessing amounts to approximately 3% of the per-run training FLOPs budget (the exact figure depends on average rollout length, which is shorter for prefixed episodes). (3) The breakeven point: amortization holds once the same model+data combination is used for 2 or more training runs (e.g., a seed sweep or hyperparameter search), which is the typical use case in RL post-training. For a single run, the overhead is small but nonzero. We will also add a brief qualification to the abstract ('at matched training compute, excluding one-time per-model calibration') so that the claim is not overgeneralized. The conservative-constant cold-start variant (Table 3, 46.4), which skips the calibration sweep and retains only the difficulty probe, provides a lower-overhead fallback that we will mention alongside the cost disclosure. revision: yes

Circularity Check

0 steps flagged

No circularity found: the 50% set-point is adopted from external prior work, evaluation is at ρ=0, and the controller mechanism is independently testable against fixed-prefix baselines.

full rationale

The paper's central claim — that closed-loop control of prefix length improves GRPO signal on hard problems at matched FLOPs — does not reduce to its inputs by construction. The 50% success-rate set-point (τ=0.5) is explicitly adopted from external prior work [Bae et al., 2025; Gao et al., 2025], not derived by the authors. The controller mechanism (secant/binary-search root-finding on prefix ratio to track k/G) is a genuine algorithmic contribution whose effectiveness is tested against externally-defined baselines (PrefixRL [Setlur et al., 2026], Prefix-RFT [Huang et al., 2025]) under matched compute. Evaluation is at ρ=0 (no prefix), so gains are measured on the deployed policy, not on the prefix-conditioned policy used during training. The ablation in Table 3 isolates closed-loop control from fixed prefixing using the same prefix source and FLOPs budget. The target sweep (τ ∈ {0.125, ..., 0.875}) provides independent empirical evidence that the intermediate set-point matters, rather than defining the result into existence. The FLOPs accounting convention (excluding calibration sweep and difficulty probe as amortized preprocessing) is a fairness assumption that could be questioned on correctness grounds, but it is not a circularity: the training-time comparison itself is not constructed to produce the claimed result by definition. No self-citation chain is load-bearing for the central claim — the key prior results invoked (intermediate set-point, back-generalization from prefixing) come from independent author groups. The paper is self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

7 free parameters · 4 axioms · 0 invented entities

No new entities, particles, or forces are introduced. The method is a control algorithm operating on existing RL training infrastructure. The free parameters are hyperparameters of the controller, not fundamental constants. The axioms are domain assumptions about GRPO dynamics and prefix-conditioned success rates, all of which are empirically tested within the paper.

free parameters (7)
  • τ (target success rate) = 0.5
    Adopted from prior work [Bae et al., Gao et al.]; empirically validated in ablation (Table 3: 0.375–0.625 all within CI)
  • w (annealing start fraction) = 0.8
    Fraction of training before envelope begins decaying; chosen by the authors
  • δ offset range = ±0.15
    Hand-tuned slope for per-sample difficulty offset around the base prefix; the paper acknowledges this is a static heuristic
  • N (controller update interval) = 10
    Optimizer steps between controller updates; not swept extensively
  • β (EMA smoothing) = 0.7
    Smoothing factor for window-mean k/G; not ablated
  • ρ_max (max prefix ratio) = not explicitly stated
    Upper bound on prefix ratio; used in clipping
  • conservative cold-start constant = 0.8
    Alternative to calibration sweep; over-prefixes to ensure k≥1
axioms (4)
  • domain assumption κ(ρ) is near-monotone in ρ
    Section 3, 'Prefix as a difficulty dial': the conditional success rate rises near-monotonically with prefix length. Violated on ~6% of measured pairs (Section 5.4); the controller falls back to bracketing in those cases.
  • domain assumption 50% success rate maximizes GRPO learning signal
    Section 4.1: adopted from [Bae et al., 2025; Gao et al., 2025]. Two intuitions given (p(1-p) scaling, k(G-k) contrastive pairs) but explicitly stated as not proofs. Empirically supported by target sweep (Table 3).
  • domain assumption Back-generalization: training with prefixes transfers to no-prefix performance
    Section 5.4: the structural premise that skills learned on partial solutions transfer to unaided solving. Tested directly via ρ=0 evaluation on originally-dead problems (~17% solved at pass@1).
  • domain assumption FLOPs = 2ND_samp + 6ND_upd accurately captures training compute
    Section 4, Compute accounting: follows [Setlur et al., 2026; Snell et al., 2024]. Calibration and difficulty probe excluded as amortized preprocessing.

pith-pipeline@v1.1.0-glm · 16643 in / 4754 out tokens · 458343 ms · 2026-07-09T02:43:19.038644+00:00 · methodology

0 comments
read the original abstract

Group Relative Policy Optimization (GRPO) stalls on a model's hardest problems: when no rollout in a group succeeds, the group-relative advantages vanish and the problem contributes no gradient, wasting the frontier examples we most want to learn from. Prepending a correct prefix of a reference solution raises the success rate, making prefix length a continuous knob on difficulty. Concurrent methods set the knob once; AdaPrefix-GRPO turns it into a feedback controller: throughout training it adjusts how much of the solution each problem gets, holding its success rate near 50%, where GRPO's gradient signal is largest, then withdraws the assistance entirely, so the deployed model solves problems unaided. On hard math, at matched training FLOPs, it more than doubles GRPO's accuracy on held-out problems from the training distribution for a 0.6B model (2.1x), with 1.6x on Qwen3-1.7B and 1.7x on AIME, while roughly halving trace length. The method is implemented in data preparation plus a loss mask on prefix tokens; the trainer is otherwise stock. The smaller the model, the larger the gain.

Figures

Figures reproduced from arXiv: 2607.07674 by Vladislav Beliaev.

Figure 1
Figure 1. Figure 1: Held-out pass@1 (no prefix) versus cumulative training FLOPs on hard math, Qwen3-1.7B. Both [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The mechanism. (a) Prefix length is a dial: the conditional success rate κ(ρ) rises near￾monotonically from the base rate to ≈ 1, so a unique ρ ⋆ realizes a reachable target τ . (b) Closed-loop control of the base prefix (with per-sample offsets) holds the batch-mean k/G at τ = 0.5 (band: 95% CI) while PrefixRL’s static lengths drift toward k =G and leave the informative regime. Once annealing starts and t… view at source ↗
Figure 3
Figure 3. Figure 3: Separating the controller from the revisit effect (Qwen3-1.7B, DeepMath held-out 250-problem [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Control in action, by difficulty cohort (Qwen3-1.7B). Problems are split into quintiles by cold-start [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Distribution of per-sample k/G at mid-training (Qwen3-1.7B). Base control with per-sample offsets concentrates mass near the τ = 0.5 set-point, where signal is maximal; a fixed prefix has drifted toward saturation, with most problems piled at k/G= 1 and a hard core stuck at k/G= 0, since its lengths were tuned once at the start; most of its rollouts thus fall in the two zero-signal modes. back-generalizati… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

21 extracted references · 21 canonical work pages · 16 internal anchors

  1. [1]

    DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

    DeepSeek-AI. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948,

    work page internal anchor Pith review Pith/arXiv arXiv

  2. [2]

    Setlur, Z

    A. Setlur, Z. Wang, A. Cohen, P. Rashidinejad, S. M. Xie. Reuse your FLOPs: Scaling RL on Hard Problems by Conditioning on Very Off-Policy Prefixes (PrefixRL). arXiv:2601.18795,

    work page arXiv

  3. [3]

    Y. Qu, A. Setlur, V. Smith, R. Salakhutdinov, A. Kumar. POPE: Learning to Reason on Hard Problems via Privileged On-Policy Exploration. arXiv:2601.18779,

    work page arXiv

  4. [4]

    Blending Supervised and Reinforcement Fine-Tuning with Prefix Sampling

    Z. Huang, T. Cheng, Z. Qiu, Z. Wang, Y. Xu, E. M. Ponti, I. Titov. Blending Supervised and Reinforcement Fine-Tuning with Prefix Sampling (Prefix-RFT). arXiv:2507.01679,

    work page internal anchor Pith review Pith/arXiv arXiv

  5. [5]

    Zhang, S

    X. Zhang, S. Wu, Y. Zhu, H. Tan, S. Yu, Z. He, J. Jia. Scaf-GRPO: Scaffolded Group Relative Policy Optimization for Enhancing LLM Reasoning. arXiv:2510.19807,

    work page arXiv

  6. [6]

    arXiv:2503.23905. S. Bae, J. Hong, M. Y. Lee, H. Kim, J. Nam, D. Kwak. Online Difficulty Filtering for Reasoning Oriented Reinforcement Learning. arXiv:2504.03380,

    work page internal anchor Pith review Pith/arXiv arXiv

  7. [7]

    Z. Gao, J. Kim, W. Sun, T. Joachims, S. Wang, R. Y. Pang, L. Tan. Prompt Curriculum Learning for Efficient LLM Post-Training. arXiv:2510.01135,

    work page arXiv

  8. [8]

    arXiv:2402.05808. C. Florensa, D. Held, X. Geng, P. Abbeel. Automatic Goal Generation for Reinforcement Learning Agents. ICML,

    work page internal anchor Pith review Pith/arXiv arXiv

  9. [9]

    Proximal Policy Optimization Algorithms

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, O. Klimov. Proximal Policy Optimization Algorithms. arXiv:1707.06347,

    work page internal anchor Pith review Pith/arXiv arXiv

  10. [10]

    Tulu 3: Pushing Frontiers in Open Language Model Post-Training

    N. Lambert et al. Tülu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv:2411.15124,

    work page internal anchor Pith review Pith/arXiv arXiv

  11. [11]

    DAPO: An Open-Source LLM Reinforcement Learning System at Scale

    12 Q. Yu et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv:2503.14476,

    work page internal anchor Pith review Pith/arXiv arXiv

  12. [12]

    Z. Liu, C. Chen, W. Xu, P. Qi, T. Pang, C. Du, W. S. Lee, M. Lin. Understanding R1-Zero-Like Training: A Critical Perspective (Dr. GRPO). arXiv:2503.20783,

    work page internal anchor Pith review Pith/arXiv arXiv

  13. [13]

    SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training

    T. Chu et al. SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-Training. arXiv:2501.17161,

    work page internal anchor Pith review Pith/arXiv arXiv

  14. [14]

    The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models

    G. Cui et al. The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models. arXiv:2505.22617,

    work page internal anchor Pith review Pith/arXiv arXiv

  15. [15]

    Learning to Reason under Off-Policy Guidance

    J. Yan et al. Learning to Reason under Off-Policy Guidance (LUFFY). arXiv:2504.14945,

    work page internal anchor Pith review Pith/arXiv arXiv

  16. [16]

    L. Ma, H. Liang, M. Qiang, L. Tang, X. Ma, Z. H. Wong, et al. Learning What Reinforcement Learning Can’t: Interleaved Online Fine-Tuning for Hardest Questions (ReLIFT). arXiv:2506.07527,

    work page arXiv

  17. [17]

    Qwen3 Technical Report

    A. Yang et al. Qwen3 Technical Report. arXiv:2505.09388,

    work page internal anchor Pith review Pith/arXiv arXiv

  18. [18]

    Gemma 4 Technical Report

    Gemma Team, Google DeepMind. Gemma 4 Technical Report. arXiv:2607.02770,

    work page internal anchor Pith review Pith/arXiv arXiv

  19. [19]

    DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning

    Z. He et al. DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset. arXiv:2504.11456,

    work page internal anchor Pith review Pith/arXiv arXiv

  20. [20]

    Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters

    C. Snell, J. Lee, K. Xu, A. Kumar. Scaling LLM Test-Time Compute Optimally Can Be More Effective than Scaling Model Parameters. arXiv:2408.03314,

    work page internal anchor Pith review Pith/arXiv arXiv

  21. [21]

    Training Verifiers to Solve Math Word Problems

    K. Cobbe et al. Training Verifiers to Solve Math Word Problems (GSM8K). arXiv:2110.14168,

    work page internal anchor Pith review Pith/arXiv arXiv