{"id":"f55cee50-84f9-48df-a1d3-02709bbae7a6","arxiv_id":"2506.13923","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":7,"one_line_summary":"Guide, a new RL method that injects hints on total failure with off-policy correction, improves math pass@1 by a few points, while the paper argues RLVR progress is mostly self-distillation.","lead":"This paper studies how reinforcement learning with verifiable rewards changes AI reasoning models, and introduces a training method, Guide, that adds hints only when a model fails every attempt. The authors find that most gains come from repackaging solutions the model already knew, and they report that Guide modestly improves math test scores over standard reinforcement learning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 1's proof is invalid: Eq. 31 replaces an expectation over y with a single trajectory, so the claimed superiority of Guide-GRPO is not established; the re-derivation needed to check if it holds after this correction.","rationale":"The reader's weakest assumption is exactly the false gradient identity in Lemma 1 and the dropped expectation in Theorem 1's proof. These are internal inconsistencies, not just disagreements with prior work: the proof itself contains Eq. 30 (the correct expectation form) and then replaces it by a single-trajectory expression in Eq. 31 without conditioning or justification. The final inequality therefore does not follow from the preceding equations, so the theoretical claim is unsupported. I agree with the reader's judgment: the empirical tables are internally consistent and the Guide method is plausible, but the advertised theory is invalid and the paper's own text identifies instability when importance weights are computed relative to the sampling distribution (Appendix D), which further undercuts the claimed off-policy correction. The concrete re-derivation test is decisive because if the exact first-order term still shows Guide's advantage under the stated positivity assumption, the theorem would survive despite the bad derivation; if not, the theorem's conclusion is not established. Since the theory is central to the paper's claims, the verdict remains REJECT, though a revised version could be salvageable.","tokens_in":22614,"tokens_out":1904,"duration_ms":15834,"concrete_test":"Re-derive Theorem 1 from Appendix B.3 without the single-trajectory replacement in Eq. 31. Specifically, compute the exact first-order term of pq(θ_t + η g) for the vanilla GRPO update g_V = Σ_{q∈U} A_q ∇θ log πθ(y_q|x_q) and the Guide update g_G, keeping the full expectation E_{y∼πθ(·|x_q)}[ I[f(y)=y*_q] ∇θ log πθ(y|x_q) ]. Then compare the two expectation terms. Also check Lemma 1 by computing ∇θ log πθ(y_q|\tilde x_q) and w_q ∇θ log πθ(y_q|x_q) on a two-token or small GRU model with x_q and \tilde x_q differing by one token; if the vectors differ in direction for some θ, the lemma is false.","verdict_should_be":"REJECT","load_bearing_attack":"The advertised theoretical guarantee (Theorem 1, Appendix B.3) rests on a demonstrably incorrect step. The proof derives pq(θ_{t+1}) = pq + η pq ⟨∇θ log πθ(y_q|x_q), g⟩ + O(η^2) for a single sampled trajectory y_q, but the correct first-order Taylor expansion of the expected success probability is pq(θ_{t+1}) = pq + η E_{y∼πθ(·|x_q)}[ I[f(y)=y*_q] ∇θ log πθ(y|x_q) ]·g + O(η^2). Replacing the expectation with one trajectory is not justified; the paper even writes Eq. 30 using the expectation and then immediately drops it in Eq. 31. The same defect then propagates: Eq. 34 states ⟨∇θ log πθ(y_q|x_q), g⟩ = [A_q + (1−p_q)^k \tilde A_q] ‖∇θ log πθ(y_q|x_q)‖², which holds only for that drawn trajectory, and the subsequent sums over q of pq times gradient norms are not equivalent to the change in the expected number of solved problems. Additionally, Lemma 1 (Eq. 16), which claims ∇θ log πθ(y_q|\tilde x_q) = w_q ∇θ log πθ(y_q|x_q) with w_q = πθ(y_q|x_q)/πθ_old(y_q|\tilde x_q), is generally false; the guidance-conditioned gradient is not a scalar multiple of the plain-prompt gradient for all parameters, so it cannot provide the stated off-policy alignment. Because these steps are load-bearing, the central claim that Guide-GRPO's learning efficiency is theoretically established is not supported. The empirical results in Tables 1–2 could still hold, but they would need to stand on the experiments alone, not on the theorem.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies how RLVR improves reasoning models, decomposing progress into self-distillation (converting pass@k reachable solutions into pass@1) and genuine capability gain (solving problems unreachable even at k attempts). The authors report that self-distillation dominates across model scales from 0.5B to 72B on a large multi-domain dataset. They then propose Guide, a family of online RL algorithms that injects natural-language hints only when all plain rollouts fail, and applies an importance-weighting correction to keep guided updates aligned with the unguided objective. They specialize Guide to GRPO and PPO, report improved math benchmark performance over vanilla GRPO and filtering baselines, and state a theoretical result (Theorem 1) claiming that Guide-GRPO improves one-step expected learning efficiency over vanilla GRPO under a positive guided-advantage assumption.","tokens_in":22996,"tokens_out":4368,"duration_ms":49242,"significance":"The empirical decomposition of RLVR progress into self-distillation versus capability gain is a valuable and clearly presented observation, backed by experiments across five model scales and more than 500,000 problems. The proposed Guide method is simple and the ablations over guidance thresholds, importance-weight choices, and PPO clipping are informative. If the theoretical guarantee were valid, it would strengthen the case that guided failure-only rollouts accelerate learning. However, the central theorem and its supporting lemmas contain serious mathematical errors, so the theoretical contribution is not established. The empirical results alone provide partial support for Guide, but the paper's advertised claim of a provable efficiency gain currently rests on incorrect derivations.","major_comments":[{"comment":"Lemma 1 is false. The claim that ∇θ log πθ(yq|x~q) = wq ∇θ log πθ(yq|xq) for a scalar wq cannot hold in general: changing the conditioning text from xq to x~q changes the full conditional distribution over token sequences, and the gradient of its log-likelihood is not a scalar multiple of the plain-prompt gradient across all parameter values. A simple counterexample is a two-token distribution where guidance changes the relative probabilities of tokens; no per-example scalar weight can align the two gradients. Since this lemma is the stated justification for the off-policy importance correction in Eq. (6) and for the gradient-alignment claim used in Theorem 1, the off-policy correction is not theoretically justified.","section":"Appendix B.2, Eq. (16)"},{"comment":"The proof of Theorem 1 replaces an expectation with a single trajectory without justification. Eq. (30) correctly writes ∇θpq = E_y[I[f(y)=y*_q] ∇θ log πθ(y|xq)], but Eq. (31) then asserts pq(θ_{t+1}) = pq + η pq ⟨∇θ log πθ(y_q|x_q), g⟩ + O(η^2) for one sampled y_q. The correct first-order term is η E_y[I[f(y)=y*_q] ∇θ log πθ(y|x_q)]·g. The subsequent Eqs. (34)-(38) all inherit this error, so the claimed expressions for E[ΔR_Vanilla] and E[ΔR_Guide] are not derived.","section":"Appendix B.3, Eqs. (30)-(31)"},{"comment":"Even apart from the missing expectation, the final comparison in Eq. (38) is not a valid consequence of the preceding equations. Eq. (37) has an outer sum over q of pq times a gradient norm, but the change in expected number of solved problems is a sum over prompts of the change in each prompt's success probability, which requires the expectation over y∼πθ(·|xq) inside each term. The proof's replacement of these expectations by single-trajectory norms makes the claimed superiority of Guide-GRPO unsupported.","section":"Appendix B.3, Eqs. (25)-(26), (37)-(38)"},{"comment":"Lemma 2's proof contains a sign error and an algebraic mistake. From Eqs. (21)-(22), the actual difference is Δ_sel − Δ_all = −η Σ_q [1−(1−p_q)^k] Ã^succ_q p_q, not −η Σ_q [1−(1−p_q)^k] (Ã^fail_q − Ã^succ_q) p_q. The printed Eq. (23) substitutes Ã^fail_q − Ã^succ_q for Ã^succ_q, which is not equivalent. Moreover, if Ã^fail_q ≥ Ã^succ_q as claimed, the sign of the correct expression is negative whenever Ã^succ_q > 0, giving Δ_sel < Δ_all, the opposite of the lemma's conclusion. Thus the claim that selective guidance outperforms or matches always-guidance is not established.","section":"Appendix B.2, Lemma 2 and Eq. (23)"},{"comment":"The text states that Guide-GRPO 'consistently outperforms all baselines' and that its improvements are 'consistent' when scaling, but the tables show several per-benchmark regressions. In Table 1, Guide-GRPO is worse than vanilla GRPO on MINERVA pass@1 (32.35 vs 32.72) and on AIME24 pass@16 (56.67 vs 60.00). In Table 2, Guide-7B-8K is worse than GRPO-7B-8K on MINERVA pass@1, AIME24 pass@1, and AIME25 pass@1. The macro-average improvements are positive, but the 'consistent' claim is not supported by the reported numbers.","section":"Section 3.2.2, Tables 1-2"}],"minor_comments":[{"comment":"The phrase 'Based of these insights' should be 'Based on these insights.'","section":"Abstract"},{"comment":"'comparsion' should be 'comparison'; 'indepedent' in Table 4's caption should be 'independent.'","section":"Appendix I.1"},{"comment":"The main text says importance ratios are re-weighted with an adaptive policy-reshaping described in Appendix E, but neither Eq. (6) nor Algorithm 1 includes the f(w_i) factor. The relationship between the reshaping and the off-policy correction should be made explicit.","section":"Appendix E and Eq. (6)"},{"comment":"Several pass@1 confidence intervals are reported as ±0.00 (e.g., GRPO MATH500 pass@1 and AIME25 pass@1 for all methods). This is surprising for quantities averaged over five independent runs and deserves a brief explanation or a different reporting method.","section":"Tables 4-5"}],"recommendation":"reject","confidential_remarks":"The empirical study is substantial and the self-distillation decomposition is a useful contribution. However, the paper's advertised theoretical guarantee is invalid: Lemma 1 is false, Lemma 2's proof has a sign error, and Theorem 1's proof drops expectations. These are load-bearing errors in the central claim of the paper, not local presentation issues. If the authors remove or substantially rewrite the theory and reframe the paper as an empirical study, a resubmission could be viable. The zero-width confidence intervals in Tables 4-5 should also be verified."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper has one genuinely useful empirical contribution: the decomposition of RLVR progress into self-distillation versus capability gain across 0.5B–72B models, on a large dataset and with confidence intervals. The finding that learning is mostly self-distillation corroborates earlier work but is more thorough. The Guide algorithm—selective hints only after total failure, with off-policy importance weighting—is a reasonable idea, and the ablations (guidance threshold, PPO-clip instability) are honest and informative.\n\nThe theory, though, does not hold. Lemma 1 (Eq. 16) asserts that the gradient of the guided-prompt log-likelihood is a positive scalar multiple of the plain-prompt gradient. That is generally false; conditioning on a different prompt changes the distribution in a way no scalar multiple can capture. Lemma 2's algebra has a sign error: subtracting \\Delta_all from \\Delta_sel gives a negative term, so the proof concludes the opposite of what the equations imply. And Theorem 1's proof replaces the expectation over sampled trajectories with a single trajectory (Eq. 30 to Eq. 31), which is unjustified and propagates into the final comparison. These are not cosmetic issues—the paper's central claim that Guide's learning efficiency is theoretically established collapses.\n\nThere are smaller issues: the decomposition labels capability gain using pass@16 while training uses 8 rollouts, and the code is not yet released, making the open-source claim hard to verify.\n\nIf you strip away the theorem, what remains is a plausible empirical method with modest but consistent gains (1–4 points) across math benchmarks, and a clean decomposition result. That is worth a serious referee, but the authors should be told to fix or remove the theoretical apparatus before publication.","headline":"Useful empirical decomposition and a plausible algorithm, but the theory is broken—Theorem 1 does not follow and should not be cited.","tokens_in":23561,"tokens_out":2965,"would_cite":false,"duration_ms":31338,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that RLVR progress on reasoning problems mostly re-weights answers the model already knows, and that injecting hints on failed attempts accelerates learning without needing hints at test time.","keywords":["reinforcement learning with verifiable rewards","self-distillation","capability gain","guidance","GRPO","importance sampling","reasoning models","math benchmarks"],"falsifier":"Compute, for a fixed guided sample $y$, the gradient $\\nabla_\\theta \\log \\pi_\\theta(y \\mid x_q)$ and $\\nabla_\\theta \\log \\pi_\\theta(y \\mid \\tilde{x}_q)$ at the same $\\theta$; if the ratio of their components is not constant across parameter directions, Lemma 1 fails and the first-order improvement of Theorem 1 is not established. A more direct empirical falsifier: train Guide-GRPO with the importance weight removed, and with the correct off-policy weight, on a fixed set of unsolved prompts; if the corrected version does not outperform the uncorrected one, the off-policy correction is not the driver.","tokens_in":22364,"feed_emoji":"🧠","tokens_out":5509,"duration_ms":54201,"temperature":0.7,"pith_summary":"Reinforcement learning with verifiable rewards (RLVR) is the engine behind modern reasoning models, but why it works is debated. This paper decomposes RLVR progress into two effects: self-distillation, where the model compresses answers it could already sample with many attempts into single-shot correct answers, and capability gain, where it learns to solve problems that were previously unreachable at any sampling budget. Across models from 0.5B to 72B parameters and half a million problems, the authors find that self-distillation dominates, with capability gain real but small. They then show that injecting a natural-language hint into the prompt on problems where all rollouts fail substantially raises pass@k, and they derive Guide-GRPO, which uses those guided rollouts with an importance-weight correction so the policy still improves on the unguided objective. The claim is that this accelerates learning: Guide-GRPO outperforms vanilla GRPO by up to 3-4 percentage points macro-average on math benchmarks, without guidance at test time.","feed_headline":"Hint-on-failure training lifts math reasoning up to 4 points","feed_subtitle":"Selective in-context hints expand what the model can sample, then distill those gains back without guidance at test time.","key_machinery":"The load-bearing object is the importance-weighted guided objective (Eq. 6), which samples guided rollouts $\\pi_{\\theta_{\\text{old}}}(\\cdot \\mid \\tilde{x}_q)$ when all plain rollouts fail and reweights each token's probability ratio by $\\pi_\\theta(r_t \\mid x_q, r_{