Pith. sign in

REVIEW 3 major objections 4 minor 23 references

Verifier-guided reinforcement learning can raise one-sample accuracy while erasing rare-but-recoverable correct answers, so pass@1 gains do not prove stronger reasoning.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 07:07 UTC pith:MCNMQU42

load-bearing objection A useful diagnostic paper with a clean seeded comparison, but the headline boundary-loss numbers are internally inconsistent across tables and need an audit before the main claim is credible. the 3 major comments →

arxiv 2607.20543 v1 pith:MCNMQU42 submitted 2026-07-12 cs.LG cs.AI

When RLVR Shrinks the Reasoning Boundary: Diagnosing Pass@k Inversion

Todd Zhou This is my paper
classification cs.LG cs.AI
keywords pass@k inversionRLVRGRPOboundary promptsmode commitmentbase anchoringcoverage collapsereasoning boundary
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that pass@k inversion is a boundary-mode commitment failure: RL with verifiable rewards improves one-sample accuracy, but on prompts where the base model has rare correct trajectories recoverable by repeated sampling, finite training groups rarely see them and the policy commits to the wrong mode before they are reinforced. Because of this, ordinary training can solve fewer distinct problems at large k than the base model—on the olympiad-level benchmark, pass@256 drops from 69.1 to 68.3 even as pass@1 rises from 10.2 to 25.1. The paper demonstrates the failure localizes to boundary prompts (654±35 base-solvable prompts lost per seed) and that a prompt-conditional base anchor, PBA, preserves those prompts (91±18 lost) and lifts pass@256 to 73.0 while also improving pass@1. If correct, this reframes what RLVR gains mean: improvement on one sample cannot be read as stronger reasoning when the reachable set has shrunk. The practical stakes are large-k inference systems—self-consistency, verifier reranking, search, and agentic repair—whose value depends on rare recoverable modes surviving training.

Core claim

The paper's central claim is that pass@k inversion—verifier-guided reinforcement learning improving one-sample accuracy while reducing many-sample coverage—is a boundary mode-commitment failure, not generic entropy collapse. On a 3000-prompt diagnostic split, ordinary GRPO loses 654±35 prompts that the frozen base model could solve under repeated sampling, while leaving pass@256 (68.3) below the base's 69.1. A deliberately simple per-prompt base anchor, PBA, protects prompts whose base correct-mode mass is too low to show up in finite RL groups; it loses only 91±18 boundary prompts and lifts pass@256 to 73.0 while also improving pass@1 by about 3.9 points over matched GRPO. The paper argues

What carries the argument

The central object is a two-mode model of each prompt as a mixture of a correct trajectory family with mass w*(x) and an incorrect family, so the one-sample base success probability equals w*(x). With an RL rollout group of size G, the probability a group contains no correct trajectory is (1 - w*)^G; for boundary prompts this is near 1, meaning updates provide no direct evidence for the correct mode before entropy falls and shared parameters drift. PBA, the Per-Problem Base Anchoring gate, is the operational probe: it estimates w*(x) from G0 frozen-base rollouts, applies ordinary GRPO when the estimate clears a threshold, and otherwise anchors the prompt to the frozen base via a token-level

Load-bearing premise

The load-bearing premise is that the frozen-base rollout counts used for both diagnosis and control—256 per prompt for regime labels and 8 for the PBA gate—are accurate enough to separate rare-but-recoverable boundary modes from out-of-reach noise; the paper itself concedes, in its limitations, that a true correct-mode mass of 1% is detected by the 8-rollout gate only 7.7% of the time, so misclassification in either direction could change the headline numbers.

What would settle it

Use the paper's own 3000-prompt diagnostic split but recompute regime labels with 1024 frozen-base samples per prompt instead of 256; if ordinary GRPO still loses about 654 base-solvable boundary prompts, the localization holds, but if the loss drops toward PBA's 91, the inversion is an artifact of noisy base-rollout estimates. A second check: train on a constructed set of prompts with known correct-mode mass w and vary the RL group size G; inversion should track the probability (1-w)^G that a group misses the correct mode.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If ordinary RLVR lifts pass@1 while pass@256 stays below the base model's, the improvement is not evidence of stronger reasoning: the set of reachable correct answers has likely shrunk.
  • Coverage loss should appear almost entirely on boundary prompts (base success below 10%, but many-sample recovery above 40%), leaving easy and out-of-reach prompts nearly unchanged—a signature that distinguishes this account from generic entropy collapse or support expansion.
  • Any verifier-guided training run should report full pass@k curves and base-to-trained transition counts, not just pass@1, because a single-number summary can hide destruction of rare recoverable modes.
  • A prompt-conditional trust region—sharpen where the frozen base shows enough correct evidence, anchor where it does not—can preserve high-budget coverage without sacrificing one-sample accuracy; PBA lifted both over matched GRPO.
  • Reasoning post-training should separate three operations: preserving rare useful modes, sharpening already-correct modes, and discovering new modes; ordinary outcome-reward RL without external exploration cannot do the third.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to replace the binary gate with a risk-weighted anchor whose strength scales as a lower confidence bound on the base correct-mode mass, which could protect more coverage at lower entropy cost than the one-success rule.
  • The same boundary-mode mechanism should appear in any repeated-attempt pipeline with automatic checkers—code repair, chart or spatial reasoning, tool-using agents—where a first-sample gain that erases rare recoverable trajectories would degrade test-time compute.
  • The theory predicts a quantitative stress test: vary the RL rollout group size G on prompts with known base correct-mass w; inversion should appear when G is small enough that (1-w)^G is close to 1 and disappear as G grows.
  • If the diagnosis holds, model evaluation for reasoning should move from single-sample accuracy to a coverage curve averaged over the base-to-trained transition; otherwise benchmark rankings can reverse under self-consistency or search.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper studies pass@k inversion after RLVR: training can improve one-sample accuracy while reducing large-k coverage. It introduces a prompt-level diagnostic that classifies prompts by frozen-base behavior into solved-easy, boundary, reachable, and out-of-reach regimes, and argues that pass@k inversion is a boundary mode-commitment failure: boundary prompts contain rare correct trajectories that disappear under finite-sample GRPO before they are reinforced. It then proposes PBA, a prompt-conditional base anchor that gates sharpening on frozen-base rollout evidence and applies a KL anchor to risky prompts. Experiments on Qwen2.5-7B with GRPO on Omni-MATH-Test and MATH500 across three seeds report that ordinary GRPO improves pass@1 from 10.2 to 25.1 but leaves pass@256 at 68.3 (below base 69.1), while PBA reaches pass@1=29.0 and pass@256=73.0. A 3000-prompt diagnostic study claims GRPO loses 654±35 base-solvable boundary prompts versus 91±18 for PBA, a 7.2× reduction. The paper explicitly states that the collapse horizon is not a closed-form property of GRPO and that PBA is a proof-of-concept intervention.

Significance. If the empirical claims hold, the paper supplies a useful diagnostic framing for a phenomenon that pass@1-only evaluation misses: RLVR can shrink the reachable set even while sharpening the top sample. The promise of prompt-level, recoverability-gated regularization is timely, and the released artifact—per-prompt counts, seed-resolved tables, matched GRPO controls, gate ablations, and run cards—is a genuine strength. The two-mode model also yields a falsifiable prediction (boundary-localized coverage loss). However, the headline quantitative support is currently undercut by an internal inconsistency in the transition tables, and the central mechanistic claim is only an endpoint signature rather than a derivation. The significance is therefore conditional on the audit and on a recalibration of the claim's strength.

major comments (3)
  1. [Appendix B, Tables 11 and 12] Table 11 reports 'All base-solved→lost' as 201±17 (seed G0=200) for GRPO, while Table 12 reports boundary-only 'Base-solved→lost' as 654±35 (seed G0=652). Boundary prompts are a subset of all prompts, so the all-regime count cannot be smaller than the boundary count. The same contradiction appears in every seed column. Since the 7.2× reduction in boundary lost prompts in Table 3 and Sec. 1 is the central empirical claim, this needs an audit: either the 'All' row uses a different definition of base-solved (e.g., pass@1) or one table is wrong. The per-prompt counts in the artifact should resolve this; as printed, the central diagnostic cannot be evaluated.
  2. [Sec. 5, Eq. (8), Proposition 1] The paper's central claim is 'pass@k inversion is a boundary mode-commitment failure,' but Eq. (8) is assumed, not derived from GRPO, and Eq. (9) merely restates the sign of pass@k change conditional on that assumption. The text concedes the 'collapse horizon is not a closed-form property of GRPO' and the Limitations say there is no checkpoint-level validation of the collapse horizon. Consequently the experiments demonstrate boundary-localized loss, but not the proposed mechanism (commitment before rare positives are reinforced); global entropy collapse or other dynamics could produce the same endpoint. Please either soften the central claim to 'consistent with' mode commitment or provide checkpoint-level trajectory evidence.
  3. [Sec. 4 and Sec. 6, Table 1] Regime labels and the PBA gate use finite frozen-base rollouts. With G0=8 and τ=0.10, a true w*=0.002, 0.01, or 0.05 has only 1.6%, 7.7%, or 33.7% probability of being detected (Table 1). The text says robustness to cutoffs is 'part of the required evidence,' but no sweep over the regime thresholds (solved/boundary/out-of-reach cutoffs) or over calibration-split noise is reported; Table 6 varies only G0 and τ. Please show that boundary lost counts and the PBA-vs-GRPO contrast are stable under these choices, or quantify the misclassification rates.
minor comments (4)
  1. [Sec. 7, Tables 3 and 12] 'Boundary entropy↑' (Table 3) and 'Entropy retained' (Table 12) are used as mechanism evidence but no definition is given. Specify how entropy is measured (e.g., rollout-level token entropy, per-prompt answer diversity, or pass@k-based) before these numbers can be interpreted.
  2. [Sec. 6, Eq. (12)] The sampled token-level KL estimator can be negative for individual rollouts. The text notes that its expectation is the forward KL, but it would help to state explicitly whether the minibatch-average regularizer is used without per-rollout clipping, since clipping could bias the estimator.
  3. [Table 5] The 'Random protection' row is a single-mask control, and the paper correctly cautions against reading it as a distributional baseline. Consider labeling it as a demonstration rather than a control to avoid over-interpretation by readers.
  4. [Sec. 7 and Appendix B] 'Base-solved→lost' is used without a single formal definition. Define whether 'base-solved' means at least one verifier-positive completion in the frozen-base 256-sample cache, and define 'lost' as zero verifier-positive completions in the trained-policy evaluation cache. This would prevent ambiguity in the transition tables.

Circularity Check

2 steps flagged

Partial circularity: PBA's boundary preservation is built into its gate; boundary-regime localization overlaps the definition, but pass@1 gains and Proposition 1's conditional derivation are independent.

specific steps
  1. self definitional [Sec. 6 (Eq. 11) and Sec. 7 (Table 3)]
    "Define a binary sharpening mask m(x)=1{bw⋆(x)≥τ}. During training, prompts with m(x)=1 receive the ordinary GRPO loss. Prompts with m(x)=0 are protected by an anchor to the base policy: LPBA=m(x)LGRPO(x)+(1−m(x))βa KL(πθ∥π0). ... The central transition is a 7.2× reduction in boundary lost prompts, from 654±35 to 91±18."

    The PBA gate is defined using the same frozen-base correct-mass estimate (bw*) that defines the boundary regime (pπ0<0.10). Prompts with bw*<τ, which include most boundary prompts, are by construction anchored to the base distribution, so their base-solvable coverage is preserved by the algorithm's definition. Reporting this preservation as the expected signature is therefore a restatement of the intervention, not an independent test of the mechanism. The independent content is the simultaneous pass@1 improvement, which limits the circularity.

  2. self definitional [Sec. 4 regime definitions and Sec. 5 Prop. 1 (Eqs. 8–9)]
    "Boundary: pπ0(x)<0.10 and 1−(1−pπ0(x))256 >0.4 ... πcom(· |x) = p−(· |x), w⋆(x)<τ ... [the second branch] reduces it for every k≥1 when w⋆(x)>0."

    The boundary regime is defined as low correct-mode mass (pπ0<0.10) with high base pass@256, i.e., rare-but-recoverable modes. Proposition 1 predicts loss exactly for w*<τ; with τ=0.10 (the value used in the PBA gate and equal to the boundary cutoff), the predicted-loss set coincides with the boundary set. Thus the localization of loss to boundary prompts is partly baked into the regime definition rather than independently derived from GRPO dynamics. The measured magnitudes are not forced by this, so the step is partially -- not wholly -- circular.

full rationale

The paper is not broadly circular. Proposition 1 (Eq. 9) is a conditional mathematical identity deriving the sign of pass@k change from the explicitly stated mode-commitment endpoint (Eq. 8); the paper does not fit it to outcomes and acknowledges it is an endpoint abstraction. The GRPO diagnostic counts (654±35 boundary lost) are empirical measurements, and PBA's pass@1 improvement over GRPO is an independent outcome. The circularity is confined to two definitional overlaps. First, PBA is deliberately designed to anchor prompts with low frozen-base correct evidence, which are exactly the boundary prompts; hence 'PBA preserves boundary prompts' is a designed consequence of the gate, not an independent validation of the mechanism. Second, the boundary bin (pπ0<0.10, high base pass@256) overlaps the theory's predicted-loss condition (w*<τ for τ=0.10), so the localization claim is partially baked into the bin definition. These overlaps are partial: the magnitudes (654 vs 91, -39 vs +1.7) and the pass@1 contrast are not forced by definitions. There are no load-bearing self-citations; all references are to external prior work. One additional issue noted but not counted as circularity: Table 11 reports All base-solved→lost as 201±17 while Table 12 reports boundary-only base-solved→lost as 654±35, an apparent internal inconsistency that warrants audit as a correctness concern.

Axiom & Free-Parameter Ledger

6 free parameters · 7 axioms · 3 invented entities

The central claim rests on: hand-set regime and gate thresholds, an idealized two-mode rollout model with a perfect verifier, the unmeasured collapse-horizon assumption, and the assumption that small frozen-base samples identify boundary prompts. Some are explicitly flagged by the authors as abstractions; none are fitted to the target outcomes, so the circularity burden stays moderate.

free parameters (6)
  • regime thresholds (solved_easy, boundary, out_of_reach) = 0.6 / 0.10 / 0.4 / 0.4
    Hand-set in §4 to operationalize regimes; the paper says thresholds 'are not fundamental constants' and asserts robustness to nearby cutoffs is 'part of the required evidence,' but no cutoff-sweep table appears in the main text.
  • PBA gate threshold τ = 0.10 (effective 0.125 for G0=8 one-success rule)
    §6, Algorithm 1; chosen so one verifier-positive base rollout suffices to sharpen. Table 6 reports G0=16/32 and τ=0.20 variants; no tuned sweep for βa or refresh period.
  • G0 (frozen-base rollouts per prompt for gate) = 8
    §7; chosen to match GRPO rollout group G=8 and keep overhead at 1.0% of policy rollouts; G0=32 improves performance in single-run Table 6, so the headline is a low-cost default rather than an optimum.
  • anchor KL weight βa = 1.0
    Eq. (11); set without reported sensitivity analysis; the paper lists matched single-setting controls only.
  • cache refresh period = 100 steps
    §7, Algorithm 1; implementation choice; the paper says 'We treat cache refresh as an implementation choice rather than independent evidence.'
  • effective mode-commitment threshold τ (Eq. 8) = not estimated
    Theoretical construct in the two-mode model; the paper explicitly does not estimate a calibrated dynamics risk score (§5).
axioms (7)
  • domain assumption The verifier returns a perfect binary label: r(x,y) ∈ {0,1}, E_{p+}[r]=1, E_{p-}[r]=0 (Eq. 6).
    Assumed in §5 for the two-mode model; the paper lists verifier noise and partial credit as abstractions in §8 Limitations.
  • ad hoc to paper Each prompt's base distribution is an ideal two-mode mixture: π0 = w* p+ + (1-w*) p- (Eq. 5).
    Idealized model; the paper calls it 'the idealized binary case' and notes real prompts have multiple correct families.
  • ad hoc to paper An all-zero GRPO group contributes no reward gradient (A_i = 0 'up to implementation details').
    §5 case 1 (Eq. 7 and following); true for group-normalized relative advantages when all rewards equal, but the model treats implementation details as negligible without evidence.
  • ad hoc to paper Shared-parameter updates from other prompts drift boundary prompts before rare positives are reinforced ('collapse horizon').
    §5; the paper states 'This horizon is not a closed-form property of GRPO' and does not measure it.
  • standard math Finite-sample pass@k estimator (Eq. 3) is unbiased for fixed n.
    Standard hypergeometric estimator from Chen et al. [2]; used for all pass@k values.
  • domain assumption 256 frozen-base completions give regime labels and base pass@k values accurate enough for diagnosis.
    §4/§7; boundary bins rely on p̂0 with resolution 1/256; no confidence intervals on regime labels are reported (bootstrap scripts are in the artifact).
  • domain assumption Forward policy-sampled token-level KL (Eq. 12) is a safe early anti-drift regularizer.
    Used in PBA loss Eq. (11); the paper itself notes that once a mode has collapsed, forward KL on policy samples cannot restore it, so the anchor only works pre-collapse.
invented entities (3)
  • Boundary regime (diagnostic bin) independent evidence
    purpose: To localize pass@k inversion to prompts with low one-sample success but recoverable high-k coverage.
    Operationalized from frozen-base rollouts in §4; any practitioner can compute it on new models, so it is measurable outside this paper. But it is a construct, not a measured physical object.
  • Effective collapse horizon no independent evidence
    purpose: To name the unknown time by which retained entropy falls before rare correct modes are reinforced.
    Introduced in §5; the paper explicitly declines to estimate it ('We therefore do not estimate a calibrated dynamics risk score') and lists checkpoint-level validation as future work.
  • Mode-commitment policy π_com (Eq. 8) no independent evidence
    purpose: To formalize the endpoint sign of pass@k change under idealized commitment.
    The paper states it is an endpoint abstraction, not a claim that GRPO produces Eq. (8).

pith-pipeline@v1.3.0-alltime-deepseek · 12025 in / 24440 out tokens · 224518 ms · 2026-08-02T07:07:55.358695+00:00 · methodology

0 comments
read the original abstract

Reinforcement learning with verifiable rewards (RLVR) can improve one-sample accuracy while making a model worse under repeated sampling. We study this pass@k inversion: after training, the policy may solve fewer distinct problems than its base model at large $k$. The failure concentrates on boundary prompts, where the base model contains rare correct trajectories that are recoverable by sampling but too sparse to reliably appear in finite RLVR rollout groups. We argue that a two-mode account explains this as an absence-of-evidence failure: rare correct trajectories may disappear before RLVR samples and reinforces them often enough. The main contribution is this diagnostic and mechanistic framing. Per-Problem Base Anchoring (PBA) is a deliberately simple proof-of-concept: sharpen prompts with sufficient frozen-base correct evidence, and anchor risky prompts to the base distribution. Across three training seeds on Omni-MATH-Test, with MATH500 as a secondary high-coverage validation benchmark, PBA improves both \PassK{1} and high-budget coverage over matched GRPO. A 3000-prompt regime-controlled diagnostic study is consistent across seeds with the expected signature: ordinary GRPO loses base-solvable boundary prompts, while PBA preserves rare verifier-positive trajectories. We use mathematical verifiers as a controlled testbed for verifier-guided optimization; the same pass@k inversion risk applies to ECCV-relevant vision-language agents when repeated visual, spatial, or chart-reasoning attempts are checked by external tools or verifiers. Reasoning post-training should decide not only how strongly to optimize, but which prompts are safe to optimize.

Figures

Figures reproduced from arXiv: 2607.20543 by Todd Zhou.

Figure 1
Figure 1. Figure 1: Omni-MATH-Test pass@k inversion. Left: mean pass@k over seeds 0– 2; bands show one seed standard deviation for trained policies and the dashed line marks base pass@256. Right: seed-paired PBA-GRPO gains with seed-SD error bars. MATH500 appears in Tab. 2 as a saturated sanity check [PITH_FULL_IMAGE:figures/full_fig_p010_1.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

23 extracted references · 17 linked inside Pith

  1. [1]

    Back to basics: Revisiting REINFORCE style optimization for learning from human feedback in LLMs

    Arash Ahmadian, Chris Cremer, Matthias Galle, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Ustun, and Sara Hooker. Back to basics: Revisiting REINFORCE style optimization for learning from human feedback in LLMs. arXiv:2402.14740, 2024

    Pith/arXiv arXiv 2024

  2. [2]

    Evaluating large language models trained on code.arXiv:2107.03374, 2021

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, and others. Evaluating large language models trained on code.arXiv:2107.03374, 2021

    Pith/arXiv arXiv 2021

  3. [3]

    The entropy mechanism of reinforcement learning for reasoning models.arXiv:2505.22617, 2025

    Ganqu Cui et al. The entropy mechanism of reinforcement learning for reasoning models.arXiv:2505.22617, 2025

    Pith/arXiv arXiv 2025

  4. [4]

    Omni-MATH: A universal olympiad level mathematic benchmark for large language models.arXiv:2410.07985, 2024

    Bofei Gao et al. Omni-MATH: A universal olympiad level mathematic benchmark for large language models.arXiv:2410.07985, 2024

    Pith/arXiv arXiv 2024

  5. [5]

    Soft actor- critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor

    Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor- critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. InICML, 2018

    2018

  6. [6]

    Measuring mathematical problem solving with the MATH dataset.arXiv:2103.03874, 2021

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset.arXiv:2103.03874, 2021

    Pith/arXiv arXiv 2021

  7. [7]

    REINFORCE++: A simple and efficient approach for aligning large language models.arXiv:2501.03262, 2025

    Jian Hu. REINFORCE++: A simple and efficient approach for aligning large language models.arXiv:2501.03262, 2025

    Pith/arXiv arXiv 2025

  8. [8]

    Reinforcement learning with KL penalties is better viewed as Bayesian inference

    Tomasz Korbak, Ethan Perez, and Christopher Buckley. Reinforcement learning with KL penalties is better viewed as Bayesian inference. InFindings of EMNLP, 2022

    2022

  9. [9]

    Conservative Q- learning for offline reinforcement learning

    Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q- learning for offline reinforcement learning. InNeurIPS, 2020

    2020

  10. [10]

    Tulu 3: Pushing frontiers in open language model post- training.arXiv:2411.15124, 2024

    Nathan Lambert et al. Tulu 3: Pushing frontiers in open language model post- training.arXiv:2411.15124, 2024

    Pith/arXiv arXiv 2024

  11. [11]

    ReMax: A simple, effective, and efficient reinforcement learning method for aligning large language models.arXiv:2310.10505, 2023

    Ziniu Li, Tian Xu, Yushun Zhang, Zhihang Lin, Yang Yu, Ruoyu Sun, and Zhi- Quan Luo. ReMax: A simple, effective, and efficient reinforcement learning method for aligning large language models.arXiv:2310.10505, 2023

    Pith/arXiv arXiv 2023

  12. [12]

    Let’s verify step by step.arXiv:2305.20050, 2023

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step.arXiv:2305.20050, 2023

    Pith/arXiv arXiv 2023

  13. [13]

    Understanding R1-Zero-like training: A critical perspective

    Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding R1-Zero-like training: A critical perspective. arXiv:2503.20783, 2025

    Pith/arXiv arXiv 2025

  14. [14]

    Manning, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. InNeurIPS, 2023. 18 Todd Y. Zhou

    2023

  15. [15]

    Vanishing gradients in reinforcement finetuning

    Noam Razin et al. Vanishing gradients in reinforcement finetuning. InICLR, 2024

    2024

  16. [16]

    Proximal policy optimization algorithms.arXiv:1707.06347, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv:1707.06347, 2017

    Pith/arXiv arXiv 2017

  17. [17]

    DeepSeekMath: Pushing the limits of mathematical reasoning in open language models.arXiv:2402.03300, 2024

    Zhihong Shao et al. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models.arXiv:2402.03300, 2024

    Pith/arXiv arXiv 2024

  18. [18]

    HybridFlow: A flexible and efficient RLHF framework

    Guangming Sheng et al. HybridFlow: A flexible and efficient RLHF framework. arXiv:2409.19256, 2024

    Pith/arXiv arXiv 2024

  19. [19]

    Qwen2.5 technical report.arXiv:2412.15115, 2024

    An Yang et al. Qwen2.5 technical report.arXiv:2412.15115, 2024

    Pith/arXiv arXiv 2024

  20. [20]

    DAPO: An open-source LLM reinforcement learning system at scale.arXiv:2503.14476, 2025

    Qiying Yu et al. DAPO: An open-source LLM reinforcement learning system at scale.arXiv:2503.14476, 2025

    Pith/arXiv arXiv 2025

  21. [21]

    Understanding diversity collapse in RLVR via the lens of over- training.arXiv:2606.15455, 2026

    Suqin Yuan et al. Understanding diversity collapse in RLVR via the lens of over- training.arXiv:2606.15455, 2026

    arXiv 2026

  22. [22]

    Does reinforcement learning really incentivize reasoning capacity in LLMs beyond the base model?arXiv:2504.13837, 2025

    Yang Yue et al. Does reinforcement learning really incentivize reasoning capacity in LLMs beyond the base model?arXiv:2504.13837, 2025

    Pith/arXiv arXiv 2025

  23. [23]

    Echo chamber: RL post-training amplifies behaviors learned in pretraining.arXiv:2504.07912, 2025

    Rosie Zhao, Alexandru Meterez, Sham Kakade, Cengiz Pehlevan, Samy Jelassi, and Eran Malach. Echo chamber: RL post-training amplifies behaviors learned in pretraining.arXiv:2504.07912, 2025

    Pith/arXiv arXiv 2025