{"id":"4c993ac2-232f-4934-9d69-7023a4fef950","arxiv_id":"2506.03106","paper_version":7,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"By training LLMs on both initial responses and critique-guided refinements, Critique-GRPO improves Pass@1 by roughly 3.8 to 6.4 points over GRPO across eight reasoning tasks.","lead":"This paper proposes Critique-GRPO, a reinforcement learning method that lets large language models learn from both right-or-wrong scores and written critiques of their failed answers. It reports consistent accuracy gains over score-only RL on eight math and science reasoning benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The default critique channel leaks ground-truth answers and solution sketches, so the headline gains and the +16.7% AIME 'self-improvement' result may largely reflect answer leakage rather than critique quality or localization.","rationale":"The reader's weakest assumption focuses on whether CoT critiques accurately localize the first error, which is important for the theoretical narrative in Proposition 4.1 and Appendix F.4. I agree that localization accuracy is not validated, but I find a more basic and more load-bearing confound: the actual critique channel in the default experiments includes the ground-truth answer, and the illustrated CoT critiques also include a correct solution sketch. Because the refinement prompt conditions on this text, the model can produce correct refinements by extracting the answer rather than by understanding and fixing its own error. This directly affects the central empirical claim, not just the theory. The paper is transparent about prompts and even reports a no-GT self-evaluation variant, which serves as internal evidence that removing the reference substantially shrinks the gain. The concrete test I propose is therefore to redact answers and solution hints from the default CoT critiques while keeping critique length and binary labels roughly matched. If the gains persist, the central claim survives; if they shrink to the no-GT level, the headline should be reframed as answer-assisted refinement rather than natural-language-critique-driven self-improvement. This does not amount to rejecting the method: code and models are released, evaluations use external benchmarks, and the within-paper ablations show a real but smaller effect for pure self-critiquing. The verdict should remain CONDITIONAL, with the condition being a clean answer-redaction ablation and a correction of the self-improvement wording.","tokens_in":52376,"tokens_out":8853,"duration_ms":117881,"concrete_test":"Run the full Critique-GRPO pipeline with the same GPT-4o critic but redact all final answers and solution hints from the critique text (instruct the critic to identify the first erroneous step and explain why it is wrong without revealing the correct answer or the correct solution approach), and retrain the self-critique variant without any ground-truth reference. If average Pass@1 and AIME24 gains over GRPO collapse toward the roughly +2.3 and +3.3 points already observed in the no-GT self-evaluation row, then the headline +15-21.6% and +16.7% improvements are attributable to answer leakage rather than to natural-language critique quality or critique localization.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that written critiques, rather than scalar rewards, drive the gains is undercut by a confound present in the default pipeline: the critique generation prompt in Appendix N includes the Ground Truth Answer, and the CoT critique examples in the main text and Appendix E contain a 'Correct Solution Approach' section ending with the final answer (e.g., cos(2θ) = 7/25). The refinement prompt then conditions on this critique, so the policy sees the correct answer during rollouts and can copy it into the final box, after which the rule-based verifier rewards it. This is not purely natural-language critique; it is answer-in-the-loop supervision. The same issue affects the headline self-improvement claim: Table 12/Appendix H show that 'Self-Critique' uses ground-truth answers as reference, while only 'Self-Critique & Self-Evaluation' removes them. The abstract's +16.7% AIME 2024 gain over GRPO is from the with-GT version; without GT, the average gain over GRPO shrinks to roughly +2.3 points and AIME24 to +3.3 points. Thus the reader's localization concern, while relevant to Proposition 4.1, is secondary: even perfectly localized critiques may contribute little once the correct answer and solution sketch are redacted. The method may still be useful as a form of guided answer-based training, but the paper's attribution of the gains to 'natural language feedback' and 'self-improvement' is not yet established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper identifies three limitations of scalar-reward-only RL for LLM reasoning (plateaus, ineffective spontaneous self-reflection, persistent failures) and proposes Critique-GRPO, an online GRPO-based framework that trains on both initial responses and critique-guided refinements, where critiques come from rule-based heuristics, GPT-4o, or the policy itself. The authors report consistent Pass@1 gains over SFT and RL baselines across eight mathematical and scientific reasoning benchmarks with Qwen2.5-7B-Base, Qwen3-8B, Qwen2.5-Math-7B-Base, Llama-3.2-3B-Instruct, and Qwen3-32B, and they claim a +16.7% AIME 2024 self-improvement gain over GRPO via self-critiquing. The paper also gives a theoretical argument, based on the Transfer Eluder Dimension, that critique-guided refinement reduces search complexity from exponential to linear in the solution length.","tokens_in":52732,"tokens_out":3903,"duration_ms":49715,"significance":"If the central attribution claim holds, the paper makes a useful contribution: it integrates natural-language critiques into online RL, provides a broad evaluation across models and benchmarks, and includes ablations, cost analysis, and robustness checks with different critique models. The release of code and models is a concrete strength. However, the headline interpretation that the gains come from natural-language feedback rather than from leakage of the ground-truth answer through the critique channel is not yet established, and the self-improvement claim is materially weaker once the ground-truth-redacted variant is isolated. The empirical core is credible but the central claim needs a cleaner control.","major_comments":[{"comment":"The default CoT-critique pipeline leaks the correct answer and a solution sketch into the training rollouts, so the headline gains do not isolate natural-language feedback. The critique-generation prompt in Appendix N includes 'Ground Truth Answer:<Ground Truth>', and the CoT critique example in Appendix E (and repeated in Appendix O) contains a 'Correct Solution Approach' section ending with 'The correct final answer is: cos(2θ) = 7/25'. The Refinement Prompt then conditions on this critique, so the policy can copy the boxed answer and receive a positive rule-based reward. A control that redacts the final answer and solution sketch from critiques is therefore required before attributing Table 2's gains to critique quality or localization; otherwise the method is better described as answer-in-the-loop supervised refinement.","section":"Section 4.2, Appendix E, Appendix N"},{"comment":"The abstract's +16.7% AIME 2024 'self-improvement' result is taken from the with-ground-truth variant. Table 12 shows that 'Self-Critique' uses ground-truth answers as reference, while the no-reference variant 'Self-Critique & Self-Evaluation' gives only a 2.3-point average gain over R1-GRPO and a 3.3-point AIME24 gain. The manuscript should either headline the ground-truth-free result or explicitly reframe the +16.7% claim as answer-supervised refinement rather than self-improvement, since the current wording overstates what the no-leakage experiment supports.","section":"Section 5.4, Table 12, Appendix H"},{"comment":"The statistical support for the empirical claims is not verifiable as reported. The text states 'All differences are considered significant at p <0.01' but gives no test procedure, no number of evaluation runs beyond 'three runs', and no variance or confidence intervals in Tables 2, 3, 5, or 6. Given that some reported improvements are as small as +0.6 point (Table 6) and +1.8 points (Table 2), the reader cannot assess whether these differences are meaningful. Please report standard errors, bootstrap intervals, or a defined significance test with the actual p-values, and describe how the best checkpoint was selected.","section":"Section 5.1, Appendix G"},{"comment":"The theoretical sample-efficiency argument assumes exact first-error localization and uniform exploration over the reduced search space, but the implementation filters critiques only on their final binary agreement with the rule-based verifier, not on the correctness of intermediate step-level localization. As the manuscript itself acknowledges in Appendix G, this filtering does not guarantee that the CoT critique correctly identifies the first error. The applicability of Proposition 4.1's exponential-to-linear reduction to the trained system is therefore unsupported; either measure localization accuracy and show it is high, or weaken the theoretical claim to apply only to idealized critiques.","section":"Proposition 4.1, Appendix F.4, Appendix G"}],"minor_comments":[{"comment":"The training data size is inconsistently reported as a '46k subset' in Section 5.1 and a '45k subset' in Appendix G; the discrepancy should be resolved.","section":"Section 5.1 vs. Appendix G"},{"comment":"The last paragraph of Appendix O says the refinement prompt is 'detailed in Appendix O', but the refinement prompt actually appears in Appendix N; the cross-reference should be corrected.","section":"Appendix O"},{"comment":"The text says 'Table 5 (upper section) shows that Critique-GRPO consistently outperforms GRPO on all eight reasoning tasks with Llama-3.2-3B-Instruct and Qwen3-32B', but Qwen3-32B results appear only in Appendix D; consider moving the 32B result into the main table or adjusting the wording.","section":"Section 5.5, Table 5"},{"comment":"The 'CoT Critique' row with 683 tokens is listed as the default, but the main text does not clearly state how the token-length variant was controlled; adding the prompting instruction used for the 2079-token variant would aid reproducibility.","section":"Table 8"}],"recommendation":"major_revision","confidential_remarks":"The core empirical finding may survive once the leakage is removed, since the no-ground-truth self-evaluation variant in Table 12 still shows a positive but much smaller gain. The authors should be pushed to make the redaction control the centerpiece of the revision, because the current abstract and Section 5.4 claims are not supported by the evidence as presented. I would also check whether the 'Lang. Feedback' column in Table 2 is misleading given that the CoT critiques embed the ground-truth answer."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know before reading this paper. First, the proposed online RL algorithm that trains on both initial responses and critique-guided refinements is new, and the experimental breadth is real: five policy models, eight benchmarks, ablations, code and models released. Second, the headline gains are very likely inflated by answer leakage. The CoT critique prompt in Appendix N includes the Ground Truth Answer, and the example critiques in the main text and Appendix E contain a \"Correct Solution Approach\" ending with the final answer. The refinement prompt conditions on that critique, so the policy can copy the answer into its final box and get rewarded by the rule-based verifier. That is not natural-language feedback in a diagnostic sense; it is answer-in-the-loop supervision.\n\nThe paper does some things well. The three limitations of scalar-reward RL (plateaus, ineffective self-reflection, persistent failures) are documented carefully. The online integration of critiques is a genuine step beyond offline NLF methods, and the shaping function and refinement sampling are sensible engineering. The ablation in Table 6 is honest enough to show that the biggest gains come from policy shaping (+3.1) and KL removal (+1.5), not from language feedback itself (+0.6). That should have been a red flag in the paper's own narrative.\n\nThe soft spots are significant. The self-improvement claim: +16.7% on AIME24 uses self-critique with ground truth as reference. When the ground truth is removed (Self-Critique & Self-Evaluation in Table 12), the average gain over GRPO drops to about +2.3 points and AIME24 to +3.3. The paper does not headline that version. There are also no error bars or variance numbers anywhere, and the best checkpoint is selected on a curated validation set. The theoretical section is an adaptation of Xu et al.'s framework and does not address the actual leakage confound.\n\nOn balance, this is a solid engineering paper that overclaims its mechanism. The central claim—that written critiques, not scalar rewards, drive the gains—is not established with the current default setup. I would send it to peer review, but only with the condition that the authors re-run the main experiments with a critique prompt that does not include the ground truth answer, and ideally with critiques that also omit the solution sketch. If the gains survive that control, the paper is a useful contribution. As it stands, the headline numbers are not trustworthy.","headline":"The online RL integration is new and the experiments are broad, but the headline gains likely reflect ground-truth leakage in the critique prompt more than the value of natural-language feedback.","tokens_in":53197,"tokens_out":2707,"would_cite":false,"duration_ms":28647,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding written critiques to reward-only RL lifts LLM reasoning by 15–22%.","keywords":["LLM reasoning","reinforcement learning","natural language feedback","critique-guided refinement","GRPO","self-critique","chain-of-thought evaluation","reasoning benchmarks"],"falsifier":"Run Critique-GRPO with corrupted critiques — keep the final 'Conclusion: incorrect [END]' label and the ground-truth answer, but replace the internal step-by-step error analysis with random or deliberately misleading statements about the solution — and compare pass@1. If accuracy stays at the level of genuine CoT critiques, the specific content of the natural-language feedback is not doing the work; if it collapses toward the indicative-critique level, the localization content is load-bearing.","tokens_in":52202,"feed_emoji":"🧠","tokens_out":13795,"duration_ms":123387,"temperature":0.7,"pith_summary":"The paper argues that the scalar right/wrong rewards used in reinforcement-learning fine-tuning of reasoning models are the bottleneck: performance plateaus even with 8× more data, spontaneous self-reflection rarely turns a wrong answer into a right one, and a fixed ~29% of training problems stay permanently unsolved. It shows that a plateaued model can often correct a failed solution when handed a written critique, and that a step-by-step chain-of-thought critique refines 55.37% of persistently failed questions versus roughly 7% for a bare 'incorrect' signal. Building on this, it proposes Critique-GRPO, an online RL algorithm that trains the policy simultaneously on ordinary sampled answers and on critique-guided refinements of failed answers, using group-relative advantages with a shaping term that amplifies correct-but-unfamiliar refinements. Across five models and eight reasoning benchmarks, the paper reports average Pass@1 gains of 15.0–21.6% on Qwen models and 7.3% on Llama-3.2-3B-Instruct over reward-only baselines, and a +16.7% gain on AIME 2024 when the model critiques itself. If correct, the conclusion is that the information content of the feedback signal, not compute or data volume, is the lever for scaling LLM reasoning.","feed_headline":"Written critiques added to RL scores lift LLM reasoning by 15–22%","feed_subtitle":"Pairs scalar rewards with critique-guided refinement, beating reward-only RL on eight reasoning benchmarks.","key_machinery":"The engine is the two-term online objective $J_{\\text{Critique-GRPO}} = J_{\\text{init}} + J_{\\text{refi}}$ built on GRPO (Group Relative Policy Optimization). For each question the policy samples $n$ initial responses; a reward system — rule-based heuristics or a model that writes chain-of-thought critiques — scores them; when the whole group fails, each failed response is refined in-context and the refinements are re-scored, then a quality-prioritized subset is pooled with the initial responses and token-level advantages are computed over the pooled group. Refined responses use a shaped importance ratio $\\rho_t(\\theta) = \\pi_t/(\\pi_t+\\gamma)$ with $\\gamma = 0.1$, which magnifies gradient signal on correct tokens the policy currently finds unlikely, while the KL penalty is removed to permit large updates toward the refinements. The theoretical load is carried by a Transfer Eluder Dimension argument: binary rewards leave the learning problem with exponential complexity $O(|S|^L)$, whereas a critique that localizes the first wrong step reduces it to $O(L|S|)$.","core_discovery":"Critique-GRPO's central claim is that natural-language critique and scalar reward are complementary learning signals, and that one online objective can capture both: standard GRPO-style updates on initial responses plus GRPO-style updates on critique-guided refinements whose importance ratios are re-weighted by a shaping function $\\rho_t(\\theta) = \\pi_t/(\\pi_t+\\gamma)$. The paper reports that this dual objective outperforms every supervised and reward-only RL baseline it is compared against, raising average Pass@1 from 42.66% to 47.08% on Qwen2.5-7B-Base and from 64.46% to 68.26% on Qwen3-8B, and that with only 4k training prompts it beats dense-intermediate-reward methods trained on 46k prompts on Qwen2.5-Math-7B-Base. The supporting theoretical argument (Proposition 4.1) uses the Transfer Eluder Dimension to explain why this should work: a critique that localizes the first wrong step decomposes a search over $|S|^L$ candidate solutions into $L$ independent sub-problems of size $|S|$, converting an exponential sample complexity into a linear one. The same machinery is used to claim self-improvement, with the model critiquing its own answers and gaining 16.7 Pass@1 points over GRPO on AIME 2024.","pith_inferences":["If the driver is the diagnostic content of the critique rather than the extra rollouts, the same loop should extend to any domain with a verifiable reward and a language diagnosis — code with compiler/test feedback, theorem proving with proof-checker errors, planning with executor failures — none of which the paper tests.","The ablation table shows the shaping term contributes the single largest jump (+3.1 points), which suggests a purely numerical re-weighting of correct-but-rare completions might recover much of the gain; a direct comparison against such a scheme would isolate how much the language content itself is worth.","Because refinement is triggered only when the entire sampled group fails, the framework implicitly spends its budget on the persistent-failure subset identified in Section 3; a curriculum that iteratively targets the surviving failures could push the data-efficiency result beyond 4k prompts.","The no-ground-truth self-critique variant improving over reward-only RL hints at a post-deployment adaptation loop, where external verification is unavailable and the model critiques its own answers against internal knowledge."],"forward_implications":["RL fine-tuning of reasoning models can escape scalar-reward plateaus at nearly constant budget: 4k Critique-GRPO prompts outpaced 46k-prompt reward-only methods on Qwen2.5-Math-7B-Base, with only 2–6.5% more GPU-hours than GRPO.","Gains transfer across critique providers (proprietary, open-weights, and small fine-tuned models) and across policy scales from 3B to 32B, so the result does not depend on one privileged teacher.","Self-critique closes the improvement loop: a model using its own chain-of-thought critiques gains +16.7% Pass@1 on AIME 2024 over GRPO, and even without any external ground truth it beats reward-only RL by +2.3% on average.","Weak-to-strong transfer works: refinements written by a weaker model improved a stronger policy by +12.3% average Pass@1, making critique-guided refinement a scalable supervision source.","Richer critiques consistently beat bare verdicts: CoT critiques outperform indicative ones by +1.8–2.4 points on Qwen2.5-7B-Base and +2.0–2.3 points on Qwen3-8B, matching the Section 3 finding that detailed critiques refine 55.37% of persistently failed questions versus ~7% for a minimal signal."],"supporting_citations":[{"why":"GRPO is the base algorithm Critique-GRPO extends: its group-relative advantage estimator and clipped objective provide the J_init and J_refi updates.","marker":"(Shao et al., 2024a)"},{"why":"Defines the R1-Zero/R1-GRPO numerical-feedback paradigm and the 'Aha moment' self-reflection claim that the paper tests and challenges; also supplies the main reward-only baseline.","marker":"(DeepSeek-AI et al., 2025)"},{"why":"Supplies the policy shaping function rho = pi/(pi+gamma) used to reweight refined-response gradients and the reorganized 46k OpenR1-Math training subset used in experiments.","marker":"(Yan et al., 2025)"},{"why":"Contributes the Transfer Eluder Dimension framework used in Proposition 4.1 to argue that critique-guided refinement cuts sample complexity from exponential to linear.","marker":"(Xu et al., 2025)"},{"why":"Source of the chain-of-thought critique construction and the Critique-FT baseline that Critique-GRPO's online refinement is compared against.","marker":"(Wang et al., 2025)"},{"why":"Dr.GRPO is the strongest reward-only baseline and provides the justification for omitting length normalization and reward standard deviation from the GRPO objective.","marker":"(Liu et al., 2025a)"},{"why":"PRIME-Zero is the dense-intermediate-reward baseline that Critique-GRPO outperforms while using 4k rather than 46k training prompts.","marker":"(Cui et al., 2025a)"},{"why":"Defines the Eluder dimension, the baseline complexity measure used in the theoretical sample-efficiency argument.","marker":"(Russo & Van Roy, 2013)"}],"fun_headline_variants":["Natural-language critiques + RL rewards lift LLM reasoning by 15–22%","Text critiques plus scalar rewards beat pure numbers for LLM reasoning","Self-critiquing adds 16.7% to GRPO on AIME 2024","Mix text and numeric feedback to boost LLM reasoning by up to 21.6%","Critique-GRPO: RL with textual criticism beats reward-only training"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the chain-of-thought critiques really localize the first error in a failed solution; the paper filters critiques only on whether their final verdict agrees with the automated answer checker, never on the truth of the intermediate step-level claims, so if the localization is unreliable the reported gains could come mostly from the loosened objective and the shaping reweighting rather than from the language feedback itself.","fun_headline_variants_meta":{"raw":{"variants":["Natural-language critiques + RL rewards lift LLM reasoning by 15–22%","Text critiques plus scalar rewards beat pure numbers for LLM reasoning","Self-critiquing adds 16.7% to GRPO on AIME 2024","Mix text and numeric feedback to boost LLM reasoning by up to 21.6%","Critique-GRPO: RL with textual criticism beats reward-only training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001514,"raw_usage":{"total_tokens":6131,"prompt_tokens":1068,"completion_tokens":5063,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":684,"completion_tokens_details":{"reasoning_tokens":4967}},"tokens_in":684,"tokens_out":5063,"duration_ms":35529,"temperature":1.0,"reasoning_tokens":4967,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:08:06.017159+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Critique-GRPO with corrupted critiques — keep the final 'Conclusion: incorrect [END]' label and the ground-truth answer, but replace the internal step-by-step error analysis with random or deliberately misleading statements about the solution — and compare pass@1. If accuracy stays at the level of genuine CoT critiques, the specific content of the natural-language feedback is not doing the work; if it collapses toward the indicative-critique level, the localization content is load-bearing.","supporting_citations":[],"review_version":1}