Pith. sign in

REVIEW 4 major objections 4 minor 13 cited by

Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Adding written critiques to reward-only RL lifts LLM reasoning by 15–22%.

desk verdict The online RL integration is new and the experiments are broad, but the headline gains likely reflect ground-truth leakage in the critique prompt more than the value of natural-language feedback. read the letter →

arxiv 2506.03106 v7 pith:BZ4MMIKS submitted 2025-06-03 cs.CL cs.AI

Xiaoying Zhang , Yipeng Zhang , Hao Sun , Kaituo Feng , Chaochao Lu , Chao Yang , Helen Meng This is my paper
classification cs.CLcs.AI
keywords LLMreasoningreinforcementlearningnaturallanguagefeedbackcritique-guidedrefinementGRPOself-critiquechain-of-thoughtevaluationbenchmarks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the scalar right/wrong rewards used in reinforcement-learning fine-tuning of reasoning models are the bottleneck: performance plateaus even with 8× more data, spontaneous self-reflection rarely turns a wrong answer into a right one, and a fixed ~29% of training problems stay permanently unsolved. It shows that a plateaued model can often correct a failed solution when handed a written critique, and that a step-by-step chain-of-thought critique refines 55.37% of persistently failed questions versus roughly 7% for a bare 'incorrect' signal. Building on this, it proposes Critique-GRPO, an online RL algorithm that trains the policy simultaneously on ordinary sampled answers and on critique-guided refinements of failed answers, using group-relative advantages with a shaping term that amplifies correct-but-unfamiliar refinements. Across five models and eight reasoning benchmarks, the paper reports average Pass@1 gains of 15.0–21.6% on Qwen models and 7.3% on Llama-3.2-3B-Instruct over reward-only baselines, and a +16.7% gain on AIME 2024 when the model critiques itself. If correct, the conclusion is that the information content of the feedback signal, not compute or data volume, is the lever for scaling LLM reasoning.

What carries the argument

The engine is the two-term online objective $J_{\text{Critique-GRPO}} = J_{\text{init}} + J_{\text{refi}}$ built on GRPO (Group Relative Policy Optimization). For each question the policy samples $n$ initial responses; a reward system — rule-based heuristics or a model that writes chain-of-thought critiques — scores them; when the whole group fails, each failed response is refined in-context and the refinements are re-scored, then a quality-prioritized subset is pooled with the initial responses and token-level advantages are computed over the pooled group. Refined responses use a shaped importance ratio $\rho_t(\theta) = \pi_t/(\pi_t+\gamma)$ with $\gamma = 0.1$, which magnifies gradient signal on correct tokens the policy currently finds unlikely, while the KL penalty is removed to permit large updates toward the refinements. The theoretical load is carried by a Transfer Eluder Dimension argument: binary rewards leave the learning problem with exponential complexity $O(|S|^L)$, whereas a critique that localizes the first wrong step reduces it to $O(L|S|)$.

What would settle it

Run Critique-GRPO with corrupted critiques — keep the final 'Conclusion: incorrect [END]' label and the ground-truth answer, but replace the internal step-by-step error analysis with random or deliberately misleading statements about the solution — and compare pass@1. If accuracy stays at the level of genuine CoT critiques, the specific content of the natural-language feedback is not doing the work; if it collapses toward the indicative-critique level, the localization content is load-bearing.

Watch

Extended reading notes

Core claim

Critique-GRPO's central claim is that natural-language critique and scalar reward are complementary learning signals, and that one online objective can capture both: standard GRPO-style updates on initial responses plus GRPO-style updates on critique-guided refinements whose importance ratios are re-weighted by a shaping function $\rho_t(\theta) = \pi_t/(\pi_t+\gamma)$. The paper reports that this dual objective outperforms every supervised and reward-only RL baseline it is compared against, raising average Pass@1 from 42.66% to 47.08% on Qwen2.5-7B-Base and from 64.46% to 68.26% on Qwen3-8B, and that with only 4k training prompts it beats dense-intermediate-reward methods trained on 46k prompts on Qwen2.5-Math-7B-Base. The supporting theoretical argument (Proposition 4.1) uses the Transfer Eluder Dimension to explain why this should work: a critique that localizes the first wrong step decomposes a search over $|S|^L$ candidate solutions into $L$ independent sub-problems of size $|S|$, converting an exponential sample complexity into a linear one. The same machinery is used to claim self-improvement, with the model critiquing its own answers and gaining 16.7 Pass@1 points over GRPO on AIME 2024.

Load-bearing premise

The load-bearing premise is that the chain-of-thought critiques really localize the first error in a failed solution; the paper filters critiques only on whether their final verdict agrees with the automated answer checker, never on the truth of the intermediate step-level claims, so if the localization is unreliable the reported gains could come mostly from the loosened objective and the shaping reweighting rather than from the language feedback itself.

Editorial extensions

If this is right

  • RL fine-tuning of reasoning models can escape scalar-reward plateaus at nearly constant budget: 4k Critique-GRPO prompts outpaced 46k-prompt reward-only methods on Qwen2.5-Math-7B-Base, with only 2–6.5% more GPU-hours than GRPO.
  • Gains transfer across critique providers (proprietary, open-weights, and small fine-tuned models) and across policy scales from 3B to 32B, so the result does not depend on one privileged teacher.
  • Self-critique closes the improvement loop: a model using its own chain-of-thought critiques gains +16.7% Pass@1 on AIME 2024 over GRPO, and even without any external ground truth it beats reward-only RL by +2.3% on average.
  • Weak-to-strong transfer works: refinements written by a weaker model improved a stronger policy by +12.3% average Pass@1, making critique-guided refinement a scalable supervision source.
  • Richer critiques consistently beat bare verdicts: CoT critiques outperform indicative ones by +1.8–2.4 points on Qwen2.5-7B-Base and +2.0–2.3 points on Qwen3-8B, matching the Section 3 finding that detailed critiques refine 55.37% of persistently failed questions versus ~7% for a minimal signal.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the driver is the diagnostic content of the critique rather than the extra rollouts, the same loop should extend to any domain with a verifiable reward and a language diagnosis — code with compiler/test feedback, theorem proving with proof-checker errors, planning with executor failures — none of which the paper tests.
  • The ablation table shows the shaping term contributes the single largest jump (+3.1 points), which suggests a purely numerical re-weighting of correct-but-rare completions might recover much of the gain; a direct comparison against such a scheme would isolate how much the language content itself is worth.
  • Because refinement is triggered only when the entire sampled group fails, the framework implicitly spends its budget on the persistent-failure subset identified in Section 3; a curriculum that iteratively targets the surviving failures could push the data-efficiency result beyond 4k prompts.
  • The no-ground-truth self-critique variant improving over reward-only RL hints at a post-deployment adaptation loop, where external verification is unavailable and the model critiques its own answers against internal knowledge.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper identifies three limitations of scalar-reward-only RL for LLM reasoning (plateaus, ineffective spontaneous self-reflection, persistent failures) and proposes Critique-GRPO, an online GRPO-based framework that trains on both initial responses and critique-guided refinements, where critiques come from rule-based heuristics, GPT-4o, or the policy itself. The authors report consistent Pass@1 gains over SFT and RL baselines across eight mathematical and scientific reasoning benchmarks with Qwen2.5-7B-Base, Qwen3-8B, Qwen2.5-Math-7B-Base, Llama-3.2-3B-Instruct, and Qwen3-32B, and they claim a +16.7% AIME 2024 self-improvement gain over GRPO via self-critiquing. The paper also gives a theoretical argument, based on the Transfer Eluder Dimension, that critique-guided refinement reduces search complexity from exponential to linear in the solution length.

Significance. If the central attribution claim holds, the paper makes a useful contribution: it integrates natural-language critiques into online RL, provides a broad evaluation across models and benchmarks, and includes ablations, cost analysis, and robustness checks with different critique models. The release of code and models is a concrete strength. However, the headline interpretation that the gains come from natural-language feedback rather than from leakage of the ground-truth answer through the critique channel is not yet established, and the self-improvement claim is materially weaker once the ground-truth-redacted variant is isolated. The empirical core is credible but the central claim needs a cleaner control.

major comments (4)
  1. [Section 4.2, Appendix E, Appendix N] The default CoT-critique pipeline leaks the correct answer and a solution sketch into the training rollouts, so the headline gains do not isolate natural-language feedback. The critique-generation prompt in Appendix N includes 'Ground Truth Answer:<Ground Truth>', and the CoT critique example in Appendix E (and repeated in Appendix O) contains a 'Correct Solution Approach' section ending with 'The correct final answer is: cos(2θ) = 7/25'. The Refinement Prompt then conditions on this critique, so the policy can copy the boxed answer and receive a positive rule-based reward. A control that redacts the final answer and solution sketch from critiques is therefore required before attributing Table 2's gains to critique quality or localization; otherwise the method is better described as answer-in-the-loop supervised refinement.
  2. [Section 5.4, Table 12, Appendix H] The abstract's +16.7% AIME 2024 'self-improvement' result is taken from the with-ground-truth variant. Table 12 shows that 'Self-Critique' uses ground-truth answers as reference, while the no-reference variant 'Self-Critique & Self-Evaluation' gives only a 2.3-point average gain over R1-GRPO and a 3.3-point AIME24 gain. The manuscript should either headline the ground-truth-free result or explicitly reframe the +16.7% claim as answer-supervised refinement rather than self-improvement, since the current wording overstates what the no-leakage experiment supports.
  3. [Section 5.1, Appendix G] The statistical support for the empirical claims is not verifiable as reported. The text states 'All differences are considered significant at p <0.01' but gives no test procedure, no number of evaluation runs beyond 'three runs', and no variance or confidence intervals in Tables 2, 3, 5, or 6. Given that some reported improvements are as small as +0.6 point (Table 6) and +1.8 points (Table 2), the reader cannot assess whether these differences are meaningful. Please report standard errors, bootstrap intervals, or a defined significance test with the actual p-values, and describe how the best checkpoint was selected.
  4. [Proposition 4.1, Appendix F.4, Appendix G] The theoretical sample-efficiency argument assumes exact first-error localization and uniform exploration over the reduced search space, but the implementation filters critiques only on their final binary agreement with the rule-based verifier, not on the correctness of intermediate step-level localization. As the manuscript itself acknowledges in Appendix G, this filtering does not guarantee that the CoT critique correctly identifies the first error. The applicability of Proposition 4.1's exponential-to-linear reduction to the trained system is therefore unsupported; either measure localization accuracy and show it is high, or weaken the theoretical claim to apply only to idealized critiques.
minor comments (4)
  1. [Section 5.1 vs. Appendix G] The training data size is inconsistently reported as a '46k subset' in Section 5.1 and a '45k subset' in Appendix G; the discrepancy should be resolved.
  2. [Appendix O] The last paragraph of Appendix O says the refinement prompt is 'detailed in Appendix O', but the refinement prompt actually appears in Appendix N; the cross-reference should be corrected.
  3. [Section 5.5, Table 5] The text says 'Table 5 (upper section) shows that Critique-GRPO consistently outperforms GRPO on all eight reasoning tasks with Llama-3.2-3B-Instruct and Qwen3-32B', but Qwen3-32B results appear only in Appendix D; consider moving the 32B result into the main table or adjusting the wording.
  4. [Table 8] The 'CoT Critique' row with 683 tokens is listed as the default, but the main text does not clearly state how the token-length variant was controlled; adding the prompting instruction used for the 2079-token variant would aid reproducibility.

Circularity Check

2 steps flagged · score 6.0 of 10

The headline gains and self-improvement claim reduce to ground-truth answers injected into the critique, so the natural-language-feedback attribution is circular by construction.

  1. fitted input called prediction [Appendix N (Prompt for Generating Chain-of-Thought Critique; Refinement Prompt), Appendix O (CoT critique example), Section 4.2 Steps 1-2, Appendix G (Reward Design)]
    "Prompt for Generating Chain-of-Thought Critique: ... Ground Truth Answer:<Ground Truth> ... Student’s Solution:<Generated Solution> Critique: ... Refinement Prompt: ... Critique:<Critique Content> ... Please re-answer by: ... Placing your final answer within \boxed. ... The correct final answer is: cos(2θ) = 7 25."

    The default CoT critique is generated with the ground-truth answer provided in the prompt, and the worked example in Appendix O ends with a 'Correct Solution Approach' that states the exact final answer. The refinement step conditions on this critique and requests a boxed final answer, while the reward is R(i) = is equivalent(y(i), yGT), i.e., equivalence to that same ground-truth answer. The reported refinement success and Pass@1 gains therefore measure the policy's ability to read or re-derive an answer that was already supplied as input; the output is not independent of the input. The paper's central claim that natural-language feedback, rather than the leaked answer or solution sketch, drives the improvements reduces by construction.

  2. fitted input called prediction [Abstract; Section 5.4; Appendix H (Table 12)]
    "Self-critique, where the model evaluates the correctness of its own responses using CoT critiques with ground truth answers as reference ... the unsupervised approach—Critique-GRPO (self-critique & self-evaluation)—achieves an average pass@1 improvement of 2.3% over R1-GRPO ... Notably, Critique-GRPO facilitates effective self-improvement through self-critiquing, achieving substantial gains over GRPO, e.g., a +16.7% Pass@1 improvement on AIME 2024."

    The headline +16.7% AIME 'self-improvement' claim is reported for the variant whose critiques use 'ground truth answers as reference'; the variant without ground-truth reference ('Self-Critique & Self-Evaluation') gains only 2.3% on average. Because the with-GT variant feeds the correct answer to the critic, and the refinement/verifier loop rewards final answers matching that same ground truth, the claimed improvement is enabled by answer-in-the-loop supervision rather than by self-critiquing alone. The prediction is forced by the construction of the critique input.

full rationale

The paper's raw benchmark comparisons are not circular: Pass@1 numbers on MATH-500, AIME, GPQA-Diamond, and other external tasks are independently measured, no fitted parameter is relabeled as a result, and Proposition 4.1 is a conditional complexity bound whose 'first error localization' assumption is not used to fit the measured numbers. There are no load-bearing self-citations. However, the evidence for the paper's central attribution claim—that natural-language critique content, rather than scalar rewards, drives the gains—is partially circular by construction. The default CoT critique prompt in Appendix N injects the ground-truth answer, the sample critique in Appendix O contains a 'Correct Solution Approach' ending with the exact final answer, and the refinement prompt conditions on that critique while the rule-based verifier rewards equivalence to the same ground-truth answer. The abstract's headline self-improvement result likewise comes from the 'Self-Critique' variant that uses ground-truth answers as reference, with the no-reference variant showing much smaller gains. This does not invalidate the measured improvements over GRPO, but it does reduce the specific claims about natural-language feedback and self-improvement to answer-in-the-loop construction. The paper's repeated statement that critiques are 'without expert demonstrations' does not remove this reduction, since the relevant leaked input is the final answer and solution sketch, not an expert trajectory.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central empirical claims rest on standard RL assumptions plus three hyperparameters chosen by validation. The theoretical sample-efficiency claim additionally assumes exact error localization and reward-informative feedback, neither of which is measured directly.

free parameters (3)
  • shaping coefficient gamma = 0.1
    Chosen in Appendix F.2 to maximize gradient on low-probability tokens; ablation in Table 6 assigns +3.1 points to policy shaping.
  • initial-to-refinement ratio = 7:1 (n=7, k=1)
    Chosen by experiments in Appendix F.3; lower ratios cause entropy explosion.
  • KL loss coefficient = 0.0
    KL regularization is removed; ablations show +1.5 points from this change (Table 6).
assumptions (4)
  • domain assumption There exists a verifier loss that measures alignment between a hypothesis and feedback (Assumption F.2, from Xu et al. 2025).
    Borrowed from the cited Transfer Eluder Dimension framework; needed for the sample-complexity argument.
  • domain assumption The employed critiques are reward-informative, i.e., they distinguish hypotheses more efficiently than binary rewards (Definition F.4).
    This is asserted rather than verified for the actual GPT-4o/DeepCritic critique distributions used in experiments.
  • ad hoc to paper Constructive CoT critiques localize the first error step exactly, enabling decomposition into O(L|S|) subproblems.
    Used in Proposition 4.1 and Appendix F.4; the paper does not measure first-error localization accuracy.
  • ad hoc to paper The proof assumes uniform exploration over the effective search space, giving success probability approximately 1 - exp(-M/d).
    This is a modeling assumption in Appendix F.4; real LLM exploration is not uniform.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback." pith.science (2026). https://pith.science/paper/BZ4MMIKS

@misc{pith2026250603106,
  author       = {Pith},
  title        = {Pith review of: Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BZ4MMIKS}},
  note         = {Machine review of arXiv:2506.03106}
}
read the original abstract

Recent advances in reinforcement learning (RL) using numerical rewards have significantly enhanced the complex reasoning capabilities of large language models (LLMs). However, we identify three fundamental limitations of purely numerical feedback: performance plateaus, ineffective spontaneous self-reflection, and persistent failures. We show that plateaued RL models can successfully refine failed solutions when given natural language critiques. Motivated by this, we propose Critique-GRPO, an online RL framework that integrates both natural language and numerical feedback for policy optimization. This approach enables LLMs to learn simultaneously from initial responses and critique-guided refinements, effectively internalizing the exploration benefits of both stages. Extensive experiments show that Critique-GRPO outperforms all compared supervised and RL-based fine-tuning methods, achieving average Pass@1 improvements of approximately +15.0-21.6% on various Qwen models and +7.3% on Llama-3.2-3B-Instruct across eight challenging reasoning tasks. Notably, Critique-GRPO facilitates effective self-improvement through self-critiquing, achieving substantial gains over GRPO, e.g., a +16.7% Pass@1 improvement on AIME 2024. The code and models are released at: https://github.com/zhangxy-2019/critique-GRPO

Figures

Figures reproduced from arXiv: 2506.03106 by the authors.

Figure 1
Figure 1. (a) Critique-GRPO enhances online reinforcement learning by enabling the model to learn from both initial responses and self-refinements through natural language feedback (critiques), highlighted in green, instead of relying solely on numerical feedback (scalar rewards). (b) Critique-GRPO improves the average Pass@1 score on Qwen3-8B by approximately 4.5% across eight reasoning tasks compared to GRPO. (c) Critique-G… view at source ↗
Figure 2
Figure 2. Overview of Critique-GRPO. Given a question, Critique-GRPO samples initial responses and then refines these responses via in-context learning, utilizing critiques generated by a reward system (either rule-based or model-based). These refinements are combined with the initial responses to optimize the policy within an online RL framework. A weighted advantage function, combined with policy shaping, emphasizes correct… view at source ↗
Figure 3
Figure 3. Comparison of Pass@k for self-improvement via Critique-GRPO (self-critique), R1-GRPO, SFT, and the base model Qwen3-8B on AIME24. Natural language feedback overcomes performance plateaus from numerical-only approaches [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Entropy dynamics on Qwen2.5-7B-Base. 5.6. Investigation of Policy Exploration To investigate whether models learn from valuable explo￾rations in refinements, we analyze the entropy dynamics of Qwen2.5-7B-Base in [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: RL fine-tuning dynamics of Qwen2.5-7B-Base (left) and Qwen3-8B (w/ Thinking) (right) using GRPO with numerical feedback on the validation set with varying numbers of training examples. Results. RL with solely numerical feedback frequently encounters performance plateau…
Figure 6
Figure 6. Figure 6: Impact of different reasoning behaviors on solving previously failed problems after RL fine-tuning with numerical feedback for Qwen2.5-7B-Base (left) and Qwen3-8B (right) on test tasks. spontaneous self-reflection. This underscores the unreliability of relying on spont…
Figure 7
Figure 7. Figure 7: Average contribution of reasoning behaviors to success￾ful completions of previously failed questions by Qwen3-8B-Base on Minerva-Math (in-distribution) and GPQA-Diamond (out-of￾distribution) datasets. We identify 50 previously unsolved problems from the Minerva-Math d…
Figure 8
Figure 8. Figure 8: Effect of γ on the shaping function f(x). The black dashed diagonal line indicates no shaping, f(x) = x. We adopt a shaping function f(x) = x/(x + γ)(Yan et al., 2025) (0 < γ < 1), depicted in Figure8, to reweight gradi￾ents and emphasize low-probability tokens in refi…
Figure 9
Figure 9. Figure 9: Comparison of Pass@k for self-improvement using RL fine-tuning via Critique-GRPO (self-critique), compared to methods relying on external numerical feedback (R1-GRPO), expert demonstrations (SFT), the base model Qwen3-8B. To explore the potential of Critique-GRPO in en…
Figure 10
Figure 10. Figure 10: Entropy dynamics during RL finetuning. A comparison of RL-based finetuning approaches for self-improvement on Qwen2.5- 7B-Base (left) and self-improvement through self-critiquing with Critique-GRPO on Qwen3-8B (right). To investigate policy exploration, we analyze two…
Figure 11
Figure 11. Figure 11: Comparison of response length changes during RL finetuning on Qwen2.5-7B-Base (left) and Qwen3-8B (right). Higher entropy does not always guarantee effective exploration. Unexpectedly, as shown in Figure 10b, Critique-GRPO (weaker refinement via critique with ground t…
Figure 12
Figure 12. Figure 12: Qualitative analysis of generated responses on Minerva-Math across four dimensions: logicality, factuality, conciseness, correctness, and average performance. Case Study. We present a qualitative comparison between the responses generated by the base Qwen3-8B model an…

Discussion (0). Sign in to comment.

Forward citations

Cited by 13 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. LLM-as-a-Coach: Experiential Learning for Non-Verifiable Tasks

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Training a language model by distilling a coach's written experiential knowledge beats training on a scalar rubric score for open-ended tasks, with better out-of-distribution transfer.

  2. Agent Reinforcement Learning via Pivotal-Aware Self-Feedback Retry

    cs.AI 2026-07 conditional novelty 6.0 of 10

    Local retry from the earliest erroneous turn, with pivotal credit isolation and reflection rewards, improves Pass@k and reduces interaction cost over MetaRL and GiGPO on agent and search tasks.

  3. REVES: REvision and VErification--Augmented Training for Test-Time Scaling

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    REVES augments LLM post-training by decoupling revision and verification signals from successful multi-step trajectories, reporting +6.5 point gains on LiveCodeBench over RL baselines.

  4. Credit Assignment with Resets in Language Model Reasoning

    cs.AI 2026-05 unverdicted novelty 6.0 of 10

    The paper introduces Random-Reset Policy Optimization (RRPO) and Self-Reset Policy Optimization (SRPO) that use resets to enable more precise credit assignment in RL for language model reasoning, with SRPO outperformi...

  5. PAIR: Prefix-Aware Internal Reward Model for Multi-Turn Agent Optimization

    cs.AI 2026-05 unverdicted novelty 6.0 of 10

    PAIR combines a hidden-state probe with an attention correction to deliver robust step-level rewards for GRPO-based optimization of multi-turn LLM agents, achieving high AUROC on contaminated trajectories at low cost.

  6. CPMobius: Iterative Coach-Player Reasoning for Data-Free Reinforcement Learning

    cs.CL 2026-02 conditional novelty 6.0 of 10

    CPMobius uses iterative coach-player reinforcement learning to improve mathematical reasoning in LLMs without external training data, yielding +4.9 average accuracy gains on Qwen2.5-Math-7B-Instruct.

  7. AdvJudge-Zero: Binary Decision Flips in LLM-as-a-Judge via Adversarial Control Tokens

    cs.LG 2025-12 conditional novelty 6.0 of 10

    Short low-perplexity tokens sampled from a judge's own next-token distribution flip binary "No" judgments to "Yes" at high rates on most tested LLM judges.

  8. Trust Region On-Policy Distillation

    cs.LG 2026-05 unverdicted novelty 5.0 of 10

    TrOPD stabilizes on-policy distillation for LLMs with trust-region learning, outlier estimation, and off-policy guidance, outperforming prior OPD methods on reasoning and code benchmarks.

  9. RL with Learnable Textual Feedback: A Bilevel Approach

    cs.LG 2026-05 unverdicted novelty 5.0 of 10

    Bi-NAC frames RL with textual feedback as a Stackelberg bilevel program and reports that 2B and 6B models trained this way outperform larger GRPO baselines on MATH-500 and GPQA.

  10. XRPO: Pushing the limits of GRPO with Targeted Exploration and Exploitation

    cs.LG 2025-10 conditional novelty 5.0 of 10

    XRPO extends GRPO with adaptive rollout allocation, in-context example seeding for unsolved prompts, and novelty-weighted advantages, reporting roughly 1-4% higher accuracy and faster convergence.

  11. CLPO: Curriculum Learning meets Policy Optimization for LLM Reasoning

    cs.AI 2025-09 conditional novelty 5.0 of 10

    CLPO: an online curriculum that rewrites each problem according to the model's own current accuracy and scales the KL penalty by difficulty, reporting strong but internally inconsistent gains over GRPO and DAPO.

  12. TaoSR-AGRL: Adaptive Guided Reinforcement Learning Framework for E-commerce Search Relevance

    cs.IR 2025-10 conditional novelty 4.0 of 10

    TaoSR-AGRL improves e-commerce search relevance by combining dense rule-aware reward shaping with adaptive ground-truth-guided replay in GRPO, reporting gains on Taobao's private offline and online evaluations.

  13. MoL-RL: Distilling Multi-Step Environmental Feedback into LLMs for Feedback-Independent Reasoning

    cs.CL 2025-07 conditional novelty 4.0 of 10

    MoL-RL combines MoL continual training on multi-step environmental feedback with GRPO post-training to improve LLM math and code reasoning without runtime feedback loops.

Reference graph

Works this paper leans on

32 extracted references · 24 canonical work pages · cited by 13 Pith papers

  1. [1]

    needle in a haystack

    Standard Generation – Reward-Only Learning (Eluder Dimension):With binary rewards r(a) =I(a=a ∗), the signal is sparse (a “needle in a haystack”). Observing r(a) = 0 eliminates only the specific sequence a, providing no information about the correctness of other sequences a′ ̸=a . Consequently, the agent must effectively enumerate the action space to find...

    work page

  2. [2]

    While the worst-case complexity remains dimT E(H, ℓ, ϵ)≈O(|S|L), the critique acts as a pruning signal

    Critique-Guided Refinement – Critique-Guided Learning (Transfer Eluder Dimension):We analyze two types of feedback mechanisms: • Indicative Feedback (cI, cGT): This feedback indicates failure but lacks specific error localization. While the worst-case complexity remains dimT E(H, ℓ, ϵ)≈O(|S|L), the critique acts as a pruning signal. Conditioning on the fa...

    work page

  3. [3]

    Factual Accuracy: Are all formulas, conversions, and physical principles factually accurate? Highlight any errors or misleading statements

    work page

  4. [4]

    Wang, Y ., Yue, X., and Chen, W

    URL http://papers.nips.cc/paper_f iles/paper/2024/hash/ad236edc564f3e3 156e1b2feafb99a24-Abstract-Datasets_ and_Benchmarks_Track.html. Wang, Y ., Yue, X., and Chen, W. Critique fine-tuning: Learning to critique is more effective than learning to imitate, 2025. URL https://arxiv.org/abs/25 01.17703. 13 Advancing LLM Reasoning with Natural Language and Nume...

    arXiv 2024

  5. [5]

    The correct maximum value, as derived from a proper analysis, should be 10 3

    **Incorrect Final Answer:** - The student concludes with an incorrect maximum value of p= 105 32 , which is not supported by the calculations. The correct maximum value, as derived from a proper analysis, should be 10 3 . Conclusion:wrong [END] 53 Advancing LLM Reasoning with Natural Language and Numerical Feedback Incorrect Initial Response (1/2) Okay, s...

    work page

  6. [6]

    URL https://arxiv.org/abs/2409.1 2122. Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., Zheng, C., Liu, D., Zhou, F., Huang, F., Hu, F., Ge, H., Wei, H., Lin, H., Tang, J., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Zhou, J., Lin, J., Dang, K., Bao, K., Yang, K., Yu, L., Deng, L., Li, M., Xue, ...

    arXiv 2025

  7. [7]

    Zhang, X., Peng, B., Li, K., Zhou, J., and Meng, H

    URL https://arxiv.org/abs/2503.1 8892. Zhang, X., Peng, B., Li, K., Zhou, J., and Meng, H. SGP- TOD: Building task bots effortlessly via schema-guided LLM prompting. In Bouamor, H., Pino, J., and Bali, K. (eds.),Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 13348–13369, Singapore, December 2023. Association for Computational L...

    doi:10.18653/v1/2023.findings-emnlp.891 2023

  8. [8]

    correct” and “incorrect

    Association for Computational Linguistics. doi: 10.18653/v1/2024.acl- long.107. URL https: //aclanthology.org/2024.acl-long.107/. 14 Advancing LLM Reasoning with Natural Language and Numerical Feedback Appendix A. Additional Discussion on Related Work Learning from Natural Language Feedback.Natural Language Feedback (NLF) provides textual critiques with g...

    arXiv 2024

Show all 32 references
  1. [9]

    As illustrated in Figure 8, this function is bounded between (0,1) , where x represents the token probability of the policy

    (0< γ <1 ), depicted in Figure8, to reweight gradi- ents and emphasize low-probability tokens in refined responses. As illustrated in Figure 8, this function is bounded between (0,1) , where x represents the token probability of the policy. When γ is small (i.e.,0.1), the func...

    2025

  2. [12]

    In this setting, the feedback function is simply the reward itself, fη(a) =r(a)

    Derivation of Effective Dimension (d) Case A: Reward-Only.The binary indicator problem is equivalent to a standard bandit problem with |S| L arms. In this setting, the feedback function is simply the reward itself, fη(a) =r(a) . Observing r(a) = 0 provides information only abo...

  3. [13]

    The probability of finding the unique optimal solution a∗ is equivalent to sampling the correct element from a set of sizedwithout replacement

    Probability of Success 25 Advancing LLM Reasoning with Natural Language and Numerical Feedback Let M be the computational budget. The probability of finding the unique optimal solution a∗ is equivalent to sampling the correct element from a set of sizedwithout replacement. The...

    2024

  4. [14]

    Since γ≪1 , the gradient magnitude is significantly dampened

    Suppression of Easy Tokens ( πt →1 ): limπt→1 Ψ(πt)≈ γ (1+γ)2 . Since γ≪1 , the gradient magnitude is significantly dampened. This prevents the model from overfitting to tokens it has already mastered, preserving the KL-divergence on trivial tokens

  5. [15]

    weaker refinement,

    Peak Efficiency in the Learning Zone:Solving dΨ dπ = 0 yields a maximum at πt =γ . By setting γ≈0.1 , we explicitly maximize gradients for ”correction tokens”—those that the current policy considers unlikely but plausible. This focuses the update on thereasoning gapbetween the...

    2017

  6. [17]

    Correctness of the Final Answer: Is the final numerical answer correct based on your calculations or the ground truth? If incorrect, provide the correct answer and explain the discrepancy

  7. [18]

    Verbosity: Is the reasoning path too verbose, too concise, or appropriate? Identify areas for condensation or expansion to improve clarity

  8. [20]

    Conclusion:

    Logical Coherence: Does the reasoning flow logically from one step to the next? Identify gaps in logic, missing steps, or irrelevant details that detract from the solution. End your analysis with: “Conclusion:” Provide ratings (1-5 scale) for each aspect. User: Question:<Quest...

  9. [21]

    Calculate the cosine of the angle of the axial section of the cone at the vertex which is also the apex of the cone

    A sphere is inscribed in a cone, such that the area of the sphere and the area of the base of the cone are equal. Calculate the cosine of the angle of the axial section of the cone at the vertex which is also the apex of the cone. CoT Critique Let’s analyze the student’s solut...

  10. [22]

    Wait, let me check that again

    So √ 0.84 = √ 84 10 = 2· √ 21 10 = √ 21 5 . Wait, let me check that again. √ 84 is √ 4·21 which is 2· √ 21, so yes, √ 84/10 = 2· √ 21/10 = √ 21/5. So cosA= √ 21/5. Wait, but √ 21 is approximately 4.583, so √ 21/5 is about 0.9166. Let me check if that’s correct. Since sinA is 0...

  11. [23]

    Then the product is(a+b)(a−b) =a 2 −b 2. So computea 2 −b 2: a2 = (2 √ 3)2 = 4·3 = 12 b2 = ( √ 21)2 = 21 Therefore,a 2 −b 2 = 12−21 =−9 Therefore, the product is −9 10·10 = −9 100 Wait, so the entire expression simplifies to −9 100? But let me check my steps again to make sure...

  12. [25]

    Therefore, the exact value is − 9 100, and the approximate decimal is −0.09

    So that matches. Therefore, the exact value is − 9 100, and the approximate decimal is −0.09. So that seems correct. Therefore, the answer is− 9 100. But let me check once more if there’s any mistake in the sign. Wait, when I calculated sin(A−B) , I had 2 √ 3− √ 21 10 . Since ...

  13. [26]

    Alternatively, if I think about angles: A is arcsin(0.4), which is in the first quadrant, B is arcsin(0.5) which is π/6, also first quadrant

    So that’s correct. Alternatively, if I think about angles: A is arcsin(0.4), which is in the first quadrant, B is arcsin(0.5) which is π/6, also first quadrant. So A+B is the sum of two first quadrant angles. Depending on their measures, A+B could be in the first or second qua...

  14. [27]

    Alternatively, using complex numbers or other methods? Maybe not necessary

    Let me check once more with another identity. Alternatively, using complex numbers or other methods? Maybe not necessary. Since two different approaches gave me the same answer, and numerical approximation also matches, I think it’s safe. But just to make sure, let me compute ...

  15. [28]

    The trigonometric substitution should be used more carefully, ensuring that the constraint is satisfied throughout

    **Misinterpretation of the Constraint:** - The student correctly rearranges the constraint abc+a+c=b to express b in terms of a and c, but then incorrectly assumes b= tan(α+γ) without proper justification. The trigonometric substitution should be used more carefully, ensuring ...

  16. [29]

    The identities used do not lead to a valid simplification of the expression

    **Incorrect Use of Trigonometric Identities:** - The student attempts to use trigonometric identities to simplify the expression for p, but the application is incorrect. The identities used do not lead to a valid simplification of the expression

  17. [30]

    The derivative should be taken with respect to the correct variables, and the critical points should be found accurately

    **Incorrect Derivative Calculation:** - The student attempts to find the maximum by taking derivatives, but the calculations are incorrect. The derivative should be taken with respect to the correct variables, and the critical points should be found accurately

  18. [31]

    The values chosen fora,b, andcdo not satisfy the constraintabc+a+c=b

    **Incorrect Evaluation of p:** - The student evaluates p at specific points, but the calculations are inconsistent and do not match the constraint. The values chosen fora,b, andcdo not satisfy the constraintabc+a+c=b

  19. [100]

    Therefore, the value of the original expression is −9 100

    So that confirms the previous result. Therefore, the value of the original expression is −9 100. But let me check once more with actual numbers to make sure. Let me computearcsin(0.4) and arcsin(0.5) numerically, then compute the sines of their sum and difference, multiply the...

  20. [150]

    Wait,

    Combined with the results in Table 2, the superior performance of Critique-GRPO over R1-Dr.GRPO and R1-GRPO highlights the importance of maintaining a certain level of entropy for better performance. Learning through self-critiquing facilitates policy exploration.Figure 10b sh...

    2025

  21. [155]

    URL https://doi.org/10.48550/arXiv .2203.02155. Qwen, :, Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., Lin, H., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Lin, J., Dang, K., Lu, K., Bao, K., Yang, K., Yu, L., Li...

    doi arXiv

  22. [2017]

    Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y ., Wu, Y ., et al

    URL https://arxiv.org/abs/1707.0 6347. Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y ., Wu, Y ., et al. Deepseekmath: Push- ing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024a. Shao, Z., W...

    doi arXiv

  23. [2024]

    Lin, T.-Y ., Goyal, P., Girshick, R., He, K., and Doll ´ar, P

    URL https://openreview.net/forum ?id=v8L0pN6EOi. Lin, T.-Y ., Goyal, P., Girshick, R., He, K., and Doll ´ar, P. Focal loss for dense object detection. InProceedings of the IEEE international conference on computer vision, pp. 2980–2988, 2017. Liu, J., Shen, D., Zhang, Y ., Dol...

    2017 arXiv

  24. [2025]

    Yan, J., Li, Y ., Hu, Z., Wang, Z., Cui, G., Qu, X., Cheng, Y ., and Zhang, Y

    URL https://arxiv.org/abs/2506.1 0341. Yan, J., Li, Y ., Hu, Z., Wang, Z., Cui, G., Qu, X., Cheng, Y ., and Zhang, Y . Learning to reason under off-policy guidance, 2025. URL https://arxiv.org/abs/ 2504.14945. Yang, A., Zhang, B., Hui, B., Gao, B., Yu, B., Li, C., Liu, D., Tu,...

    2025 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.