Pith. sign in

REVIEW 3 major objections 6 minor 38 references

RefCritic: Training Long Chain-of-Thought Critic Models with Refinement Feedback

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper shows that rewarding a critic for the fixes it produces, not just for its verdicts, yields long chain-of-thought critiques that improve policy models on five benchmarks.

desk verdict RefCritic's refinement-reward idea is genuinely new, but the headline gains may be inflated by answer leakage that is never audited; needs a leakage check before the results can be trusted. read the letter →

arxiv 2507.15024 v1 pith:IZBLJPOA submitted 2025-07-20 cs.CL

Qiaoyu Tang , Hao Xiang , Le Yu , Bowen Yu , Hongyu Lin , Yaojie Lu , Xianpei Han , Le Sun
show 1 more author
Junyang Lin
This is my paper
classification cs.CL
keywords LLMcriticschain-of-thoughtreinforcementlearningrefinementfeedbackmathematicalreasoninganswerleakageProcessBenchtest-timescaling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Supervised fine-tuning makes LLM critics good at saying whether a solution is right but bad at telling the policy model how to fix it. RefCritic replaces that with a two-reward reinforcement learning objective: one reward for correct judgments and a second reward that measures how often a policy model's revised solution reaches the ground-truth answer after following the critique. The paper reports consistent gains on five benchmarks, for example 6.8 and 7.2 percentage points on AIME25 pass@1 for Qwen2.5-14B-Instruct and DeepSeek-R1-Distill-Qwen-14B, and shows that the resulting critic outperforms step-level supervised methods on ProcessBench despite seeing only solution-level labels.

What carries the argument

The load-bearing mechanism is the dual rule-based reward: verdict reward $R_j$ equals 1 if the critic's binary judgment is correct, and refinement reward $R_r$ equals the fraction of the policy's refined solutions whose final answer matches ground truth, active only when the critic catches a genuinely incorrect solution. The objective $R_j + \lambda R_r$ is optimized with GRPO after an SFT cold start, so the model is rewarded for feedback that verifiably improves the policy rather than for confident-sounding analysis.

What would settle it

Take the trained RefCritic, strip from each critique every token that contains the final answer or a restatement of it, then measure policy pass@1 after refinement; if the gain over the SFT baseline collapses to near zero, the refinement reward was optimizing answer leakage rather than actionable analysis.

Watch

Extended reading notes

Core claim

The central discovery is that a critic's real function is to make the policy model better, and this can be trained end-to-end with rule-based rewards. RefCritic maximizes $R_j + \lambda R_r$, where $R_j$ is 1 when the critic's correctness verdict matches ground truth and $R_r$ is the fraction of refined solutions that match the final answer, paid only when the critic correctly labels an incorrect solution. This explicitly ties critique quality to downstream improvement, and in the paper it produces long chain-of-thought critiques that locate errors, identify the faulty step even on ProcessBench, and drive larger gains in both refinement and majority-vote filtering as sampling scales up.

Load-bearing premise

The load-bearing premise is that the refinement reward measures genuine helpful feedback: the critic is paid whenever the policy's revised answer matches the ground truth, and the only thing stopping the critic from simply revealing that answer is a prompt instruction saying 'Do not expose any answer!'

Editorial extensions

If this is right

  • A single round of critique then refinement lifts pass@1 on AIME25 by 6.8 points for Qwen2.5-14B and 7.2 points for DeepSeek-R1-Distill-Qwen-14B.
  • Using RefCritic to filter solutions before majority voting produces larger gains as sample counts grow, averaging +3.6 points on AIME25 at 64 samples.
  • Training on solution-level labels transfers to step-level error localization, outperforming step-supervised critics on ProcessBench.
  • Gains extend out-of-distribution to LiveCodeBench and GPQA, and RefCritic supervises stronger models such as QwQ and 32B models better than their own self-critique.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to apply the same dual-reward recipe to any domain with cheap verifiable refinement, such as code repair with unit tests or SQL generation with execution checks, since the paper only tests math-heavy tasks.
  • An immediate audit would be to mask final answers from RefCritic's critiques and measure whether the refinement gains survive; if they vanish, part of the reported improvement is answer leakage rather than actionable feedback.
  • The ProcessBench improvement suggests that solution-level reward gradients are enough to induce fine-grained error localization, which could reduce annotation cost for process-supervised models in other domains.
  • The two-stage $\lambda=0$-then-$\lambda=1$ curriculum hints that judgment should be trained before refinement, an ordering that could be tested for other reward combinations or model scales.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes RefCritic, a two-stage method for training long-chain-of-thought critic models for mathematical reasoning. Stage one is a supervised fine-tuning (SFT) cold start built from rejectively sampled critique data; stage two is a GRPO reinforcement-learning stage with two rule-based rewards: a binary judgment-accuracy reward Rj and a refinement-accuracy reward Rr that measures how often the policy model produces the correct final answer after incorporating the critique. The authors report consistent Pass@1 and majority-vote gains on AIME24, AIME25, and OlympiadBench for two 14B base models, out-of-distribution gains on LiveCodeBench and GPQA, and strong ProcessBench F1 scores despite no step-level supervision. They also argue that SFT alone is insufficient for producing actionable critiques, based on a preliminary experiment on AIME25.

Significance. If the central claims survive scrutiny, this is a significant contribution: it demonstrates a scalable, rule-based reward for training critics to give actionable feedback without step-level labels, and it reports consistent gains across multiple benchmarks and model families. The paper is commendable for using fully rule-based rewards, for reporting extensive evaluation protocols, for including ablations of the reward weighting, and for testing generalization to code, science QA, and process error detection. The dual-reward formulation is simple and reproducible, and the majority-vote scaling results are a useful practical finding. However, the significance is currently conditional because the refinement reward is vulnerable to answer leakage, an issue that is never measured, and because the ProcessBench comparison depends on an unvalidated external step extractor.

major comments (3)
  1. [Section 4, definition of Rr] The refinement reward Rr is defined as the fraction of m sampled refined solutions whose final answer matches the ground truth, and it is nonzero only when the critic correctly classifies the initial solution as incorrect. Because the critique text f is the only channel from the critic to the policy model, a critique that simply reveals the correct answer (or provides a nearly complete solution) will make almost all refined solutions match the ground truth, earning near-maximal refinement reward regardless of whether the critique contains any genuine error analysis. The paper's only safeguards are the prompt instruction 'Do not expose any answer!' in Table 6 and a rule-based leakage filter applied to the SFT seed data (Section 4); no measurement is reported of answer leakage in the final RL-trained RefCritic critiques. This is a concrete reward-hacking confound for the headline Pass@1 gains in Table 2 (e.g., 14.4 to 21.2 on AIME25 for Qwen-14B). Please audit the trained critic for leakage (e.g., by matching the ground-truth answer string or a numeric equivalent in the critique, and by human review of a random sample), and if leakage is present, add an explicit penalty or a rule-based/adversarial filter to Rr and re-report the results.
  2. [Section 5.1 and Table 3] The ProcessBench evaluation of RefCritic relies on an external extractor (Qwen2.5-14B-Instruct) to convert a natural-language critique into a step index; the paper does not validate the accuracy of this extractor. Any extraction error propagates directly into the reported F1 scores, and the comparison with step-level supervised baselines that natively output step indices (e.g., DeepCritic, ThinkPRM) may therefore be unfair or unreliable. Please provide validation of the extractor (e.g., agreement with human annotations on a sample of critiques), report the extractor's error rate, and either compare all methods under the same extraction pipeline or provide an end-to-end evaluation that does not require a separate extraction step.
  3. [Section 3 and Abstract] The preliminary claim that SFT is insufficient for deep critique is based on a single training distribution (a subset of NuminaMath, ~10K samples) and a single evaluation benchmark (AIME25), with only two base models and no variation in data scale or SFT recipe. The Abstract's statement that 'supervised fine-tuning ... fails to genuinely enhance models' critique abilities' is broader than this limited evidence supports. Please either expand the preliminary study (e.g., multiple datasets, different data scales, and additional critique benchmarks) or soften the claim so that it reads as a motivation for the RL stage rather than an established finding.
minor comments (6)
  1. [Table 4] The rows labeled 'RefCritic-R1-14B(Ours)' and 'RefCritic-Qwen-14B(Ours)' appear to be swapped between the Qwen-14B and R1-Qwen-14B blocks; please check that the results are aligned with the correct backbone model.
  2. [Figure 3 and Section 5.5] The caption of Figure 3(b) says 'AIME24' while the text in Section 5.5 describing the same figure says 'AIME25'; please reconcile the inconsistency.
  3. [Section 4, equation for Rr] The typesetting of the cases environment is corrupted: 'if c = 0andc = ˆc' should be 'if c = 0 and \hat{c} = 0', and the braces of the cases block are missing.
  4. [Notation throughout] The notation for the refinement pass@1 metric is inconsistent: 'Pass@1', 'P assr@1', and 'P ass@1' are used in different places; please unify.
  5. [Section 5.2] The sentence 'average benefit of 3.6%(1.1% for RefCritic-Qwen and 6.1% for RefCritic-R1)' should refer to 'percentage points' rather than 'percent', since these are differences in accuracy values.
  6. [References] The NuminaMath reference is cited as 'LI et al., 2024' with all-caps 'LI'; the standard formatting would be 'Li et al.'

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity found: RefCritic's training rewards are supervised signals on training data, and the reported gains are measured on held-out benchmarks rather than on quantities defined by the paper itself.

full rationale

The paper's derivation chain is not circular. The core claim is that reinforcement learning with dual rule-based rewards (judgment accuracy Rj and refinement accuracy Rr) improves both critique judgment and downstream refinement. The rewards in Section 4 are computed from ground-truth labels (c = I[y0 = a], Rj(c, c-hat), and Rr = (1/m) sum_i I[yi = a] when the critic correctly flags an incorrect solution) and are applied to NuminaMath-derived training data, while the headline results are reported on held-out benchmarks (AIME24, AIME25, OlympiadBench, GPQA, LiveCodeBench, and ProcessBench). The Pass@1-after-refinement evaluation metric is the same type of objective as the refinement reward, but it is computed on different, held-out problems; no parameter is fitted to the evaluation sets, so the reported gains are an empirical generalization claim rather than a fitted value renamed as a prediction. The paper does cite several works by overlapping authors (e.g., SCRIT, RealCritic, ProcessBench), but these are used as baselines, benchmark references, or motivational framing, not as a uniqueness theorem or as the sole justification for the central method; the SFT-insufficiency claim is supported by direct experiments in Table 1, not by citation. The reviewer-identified answer-leakage concern is a legitimate reward-hacking and robustness risk, but it is not a circularity: the reward Rr does not presuppose that the final critic avoids leaking answers, and an empirical audit of leakage would be an external check rather than a logical reduction. Overall, the paper is self-contained against external benchmarks and no load-bearing step reduces to its own inputs.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central training objective introduces one tuned weight lambda and relies on assumptions about correctness signals, answer leakage, and data curation. No new physical or architectural entities are introduced.

free parameters (1)
  • lambda (refinement reward weight) = 0 for 600 steps, then 1 for 300 steps
    Hand-chosen schedule balancing judgment accuracy and refinement utility; the authors state lambda=0 is used first for rapid critic improvement at lower cost, then lambda=1 to activate refinement feedback.
assumptions (4)
  • domain assumption Ground-truth final-answer matching via the 'math equal function' is a sufficient correctness signal for both solution judgment and refinement reward.
    Used throughout training and evaluation (Section 4, Rj and Rr definitions); may fail for equivalent but differently expressed answers, and equates refinement quality with final-answer hits rather than fixing the identified error.
  • domain assumption The prompt instruction 'Do not expose any answer!' is sufficient to prevent the critic from leaking the answer during RL.
    The refinement reward Rr rewards any critique that leads to a correct refined answer; without a rule-based leakage check, the critic could maximize reward by revealing the answer. The paper only filters leakage in SFT seed data, not in RL output (Sections 3, 4, Appendix B).
  • ad hoc to paper Difficulty filtering that excludes problems where DeepSeek-Distill-Qwen-7B solves all or none of eight samples produces a training distribution representative of test benchmarks.
    Data construction (Section 5.1 and Appendix A); these excluded problems may still be valuable for learning critique, and the threshold of eight samples is arbitrary.
  • domain assumption Using Qwen2.5-14B-Instruct as a separate extract model to convert free-form critiques into ProcessBench step indices does not systematically bias the F1 comparison against native PRM step scorers.
    ProcessBench evaluation (Section 5.4, footnote 2); no accuracy analysis of the extractor is provided, and PRM baselines marked * use their native scoring.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RefCritic: Training Long Chain-of-Thought Critic Models with Refinement Feedback." pith.science (2026). https://pith.science/paper/IZBLJPOA

@misc{pith2026250715024,
  author       = {Pith},
  title        = {Pith review of: RefCritic: Training Long Chain-of-Thought Critic Models with Refinement Feedback},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IZBLJPOA}},
  note         = {Machine review of arXiv:2507.15024}
}
read the original abstract

With the rapid advancement of Large Language Models (LLMs), developing effective critic modules for precise guidance has become crucial yet challenging. In this paper, we initially demonstrate that supervised fine-tuning for building critic modules (which is widely adopted in current solutions) fails to genuinely enhance models' critique abilities, producing superficial critiques with insufficient reflections and verifications. To unlock the unprecedented critique capabilities, we propose RefCritic, a long-chain-of-thought critic module based on reinforcement learning with dual rule-based rewards: (1) instance-level correctness of solution judgments and (2) refinement accuracies of the policy model based on critiques, aiming to generate high-quality evaluations with actionable feedback that effectively guides model refinement. We evaluate RefCritic on Qwen2.5-14B-Instruct and DeepSeek-R1-Distill-Qwen-14B across five benchmarks. On critique and refinement settings, RefCritic demonstrates consistent advantages across all benchmarks, e.g., 6.8\% and 7.2\% gains on AIME25 for the respective base models. Notably, under majority voting, policy models filtered by RefCritic show superior scaling with increased voting numbers. Moreover, despite training on solution-level supervision, RefCritic outperforms step-level supervised approaches on ProcessBench, a benchmark to identify erroneous steps in mathematical reasoning.

Figures

Figures reproduced from arXiv: 2507.15024 by the authors.

Figure 1
Figure 1. The Critic model with Refinement Feedback RefCritic framework consists of two steps: (1) cold-start via [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Supervision of RefCritic-R1-14B on stronger models like Qwen2.5-72B, DeepSeek-Distill-Qwen-32B, [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Test-time scaling with RefCritic. Subplot (a), scaling the number of critiques with RefCritic. Subplot (b), [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

38 extracted references · 5 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Zachary Ankner, Mansheej Paul, Brandon Cui, Jonathan D Chang, and Prithviraj Ammanabrolu. 2024. Critique-out-loud reward models. arXiv preprint arXiv:2408.11791

    arXiv 2024

  4. [4]

    Eugene Charniak and Mark Johnson. 2005. Coarse-to-fine n-best parsing and maxent discriminative reranking. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05), pages 173--180

    work page 2005

  5. [5]

    Jiaqi Chen, Bang Zhang, Ruotian Ma, Peisong Wang, Xiaodan Liang, Zhaopeng Tu, Xiaolong Li, and Kwan-Yee K. Wong. 2025. https://arxiv.org/abs/2504.19162 Spc: Evolving self-play critic via adversarial games for llm reasoning . Preprint, arXiv:2504.19162

    arXiv 2025

  6. [6]

    Sijia Chen, Baochun Li, and Di Niu. 2024. Boosting of thoughts: Trial-and-error problem solving with large language models. arXiv preprint arXiv:2402.11140

    arXiv 2024

  7. [7]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948

    arXiv 2025

  8. [8]

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, and 1 others. 2024. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402.14008

    arXiv 2024

Show all 38 references
  1. [9]

    Yancheng He, Shilong Li, Jiaheng Liu, Weixun Wang, Xingyuan Bu, Ge Zhang, Zhongyuan Peng, Zhaoxiang Zhang, Zhicheng Zheng, Wenbo Su, and Bo Zheng. 2025. https://arxiv.org/abs/2502.19361 Can large language models detect errors in long chain-of-thought reasoning? Preprint, arXiv...

    2025 arXiv

  2. [10]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, Kai Dang, Yang Fan, Yichang Zhang, An Yang, Rui Men, Fei Huang, Bo Zheng, Yibo Miao, Shanghaoran Quan, and 5 others. 2024. https://arxiv.org/abs/2409.12186 Qwe...

    2024 arXiv

  3. [11]

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, and 1 others. 2024. Openai o1 system card. arXiv preprint arXiv:2412.16720

    2024 arXiv

  4. [12]

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974

    2024 arXiv

  5. [13]

    Muhammad Khalifa, Rishabh Agarwal, Lajanugen Logeswaran, Jaekyeom Kim, Hao Peng, Moontae Lee, Honglak Lee, and Lu Wang. 2025. https://arxiv.org/abs/2504.16828 Process reward models that think . Preprint, arXiv:2504.16828

    2025

  6. [14]

    Tian Lan, Wenwei Zhang, Chen Xu, Heyan Huang, Dahua Lin, Kai Chen, and Xian-Ling Mao. 2024. Criticeval: Evaluating large-scale language model as critic. In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  7. [15]

    Jia LI, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. 2024. Numinamath. [https://huggingface.co/AI-MO/Numi...

    2024

  8. [16]

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let's verify step by step. arXiv preprint arXiv:2305.20050

    2023 arXiv

  9. [17]

    Zicheng Lin, Zhibin Gou, Tian Liang, Ruilin Luo, Haowei Liu, and Yujiu Yang. 2024. Criticbench: Benchmarking llms for critique-correct reasoning. arXiv preprint arXiv:2402.14809

    2024 arXiv

  10. [18]

    Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, and Yu Wu. 2025. https://arxiv.org/abs/2504.02495 Inference-time scaling for generalist reward modeling . Preprint, arXiv:2504.02495

    2025

  11. [19]

    Dakota Mahan, Duy Van Phung, Rafael Rafailov, Chase Blagden, Nathan Lile, Louis Castricato, Jan-Philipp Fr \"a nken, Chelsea Finn, and Alon Albalak. 2024. Generative reward models. arXiv preprint arXiv:2410.12832

    2024 arXiv

  12. [20]

    Nat McAleese, Rai Michael Pokorny, Juan Felipe Ceron Uribe, Evgenia Nitishinskaya, Maja Trebacz, and Jan Leike. 2024. Llm critics help catch llm bugs. arXiv preprint arXiv:2407.00215

    2024 arXiv

  13. [21]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. 2024. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling

    2024

  14. [22]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. https://arxiv.org/abs/2402.03300 Deepseekmath: Pushing the limits of mathematical reasoning in open language models . Preprint, arXiv:2402.03300

    2024 arXiv

  15. [23]

    Wenlei Shi and Xing Jin. 2025. https://arxiv.org/abs/2504.10337 Heimdall: test-time scaling on the generative verification . Preprint, arXiv:2504.10337

    2025 arXiv

  16. [24]

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314

    2024 arXiv

  17. [25]

    Zhengyang Tang, Ziniu Li, Zhenyang Xiao, Tian Ding, Ruoyu Sun, Benyou Wang, Dayiheng Liu, Fei Huang, Tianyu Liu, Bowen Yu, and Junyang Lin. 2025 a . https://arxiv.org/abs/2501.05727 Enabling scalable oversight via self-evolving critic . Preprint, arXiv:2501.05727

    2025 arXiv

  18. [26]

    Zhengyang Tang, Ziniu Li, Zhenyang Xiao, Tian Ding, Ruoyu Sun, Benyou Wang, Dayiheng Liu, Fei Huang, Tianyu Liu, Bowen Yu, and Junyang Lin. 2025 b . https://arxiv.org/abs/2501.14492 Realcritic: Towards effectiveness-driven evaluation of language model critiques . Preprint, arX...

    2025 arXiv

  19. [27]

    Qwen Team. 2025. https://qwenlm.github.io/blog/qwq-32b/ Qwq-32b: Embracing the power of reinforcement learning

    2025

  20. [28]

    Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. 2022. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.14275

    2022 arXiv

  21. [29]

    Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Y Wu, and Zhifang Sui. 2023. Math-shepherd: Verify and reinforce llms step-by-step without human annotations. CoRR, abs/2312.08935

    2023 arXiv

  22. [30]

    Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. 2024. An empirical analysis of compute-optimal inference for problem-solving with language models. arXiv preprint arXiv:2408.00724

    2024 arXiv

  23. [31]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025 a . https://arxiv.org/abs/2505.09388...

    2025 arXiv

  24. [32]

    Wenkai Yang, Jingwen Chen, Yankai Lin, and Ji-Rong Wen. 2025 b . https://arxiv.org/abs/2505.00662 Deepcritic: Deliberate critique with large language models . Preprint, arXiv:2505.00662

    2025 arXiv

  25. [33]

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36:11809--11822

    2023

  26. [34]

    Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. 2024. https://openreview.net/forum?id=CxHRoTLmPX Generative verifiers: Reward modeling as next-token prediction . In The 4th Workshop on Mathematical Reasoning and AI at NeurIPS'24

    2024

  27. [35]

    Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. 2025. The lessons of developing process reward models in mathematical reasoning. arXiv preprint arXiv:2501.07301

    2025 arXiv

  28. [36]

    Chujie Zheng, Zhenru Zhang, Beichen Zhang, Runji Lin, Keming Lu, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. 2024. Processbench: Identifying process errors in mathematical reasoning. arXiv preprint arXiv:2412.06559

    2024 arXiv

  29. [37]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, and 1 others. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595--46623

    2023

  30. [38]

    Xin Zheng, Jie Lou, Boxi Cao, Xueru Wen, Yuqiu Ji, Hongyu Lin, Yaojie Lu, Xianpei Han, Debing Zhang, and Le Sun. 2025. https://openreview.net/forum?id=JEehcb48Vp Critic-cot: Boosting the reasoning abilities of large language model via chain-of-thought critic

    2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.