Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

Reward-model benchmarks that measure correctness do not predict how well the models guide LLM reasoning.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Correctness-based reward-model benchmarks poorly predict downstream performance; best-of-N scores and generative-ability correlations are more informative.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection Useful survey with a good point about RM evaluation; treat the headline experiment as suggestive, not conclusive. the 3 major comments →

arxiv 2510.01925 v3 pith:U5QKEWGY submitted 2025-10-02 cs.CL

Enhancing Large Language Model Reasoning with Reward Models: An Analytical Survey

Qiyuan Liu , Hao Xu , Xuhong Chen , Wei Chen , Yee Whye Teh , Ning Miao This is my paper
classification cs.CL
keywords reward modelsLLM reasoningprocess reward modelsoutcome reward modelsgenerative reward modelstest-time scalingreward model evaluationreinforcement learning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This survey argues that the most common metrics used to evaluate reward models—how often they rank pairs correctly or spot an incorrect final answer—do not predict how useful they are in the two main jobs they are built for: choosing good solutions at test time and supplying training signals in reinforcement learning. The authors support this with experiments on six open-source process reward models, where a model with middling step-correctness scores ranked first in several downstream test-time tasks. They also find that generative reward models generally outperform discriminative ones, that process (step-level) models beat outcome (whole-solution) models for selecting answers but not for online RL, and that most reward models generalize poorly out of distribution. If the claim holds, practitioners should evaluate reward models by measuring their downstream effect—best-of-N accuracy, search-guiding quality, and the policy produced by RL—not by benchmark precision alone.

Core claim

The paper's central discovery is that an RM's score on correctness-style benchmarks is a weak guide to its real-world value. In their experiment, step-level correctness accuracy (the metric used by popular process-reward benchmarks) shows only a modest positive correlation with best-of-N and search-guiding scores, and the relative ranking of reward models changes depending on which policy model generates the candidate solutions. A process reward model that ranks low on correctness can rank first on multiple downstream test-time tasks. From this the paper concludes that evaluation practices need to move toward directly measuring task-level performance, while retaining process-level metrics to

What carries the argument

Reward models are learned verifiers that map a question and a reasoning trace to a scalar score, and the survey's organizing taxonomy is the double distinction between discriminative vs generative RMs (scalar-only vs critique-producing) and outcome vs process RMs (whole-solution vs step-level). The analysis that carries the argument is a comparative experiment design: same-base-model comparisons of generative versus discriminative verifiers, PRM-versus-ORM comparisons on test-time selection and online RL, and a correlation analysis plotting step-correctness accuracy against best-of-N, beam-search, and MCTS outcomes for a set of process reward models under two different policy models.

Load-bearing premise

The co-evolution claim rests on a measurement of discrimination that uses one judge prompt checking only final-answer correctness on 100 random questions per dataset; this assumes the prompt and sample represent a model's general judging ability and are not distorted by the judge's own training data or prompt sensitivity.

What would settle it

Re-run the paper's correlation experiment with a much larger set of process reward models (several dozen), two or more judge prompts, and three or more policy models. If step-correctness accuracy and downstream best-of-N/MCTS/beam scores consistently track each other (Spearman above roughly 0.8), the claim that correctness metrics are insufficient would collapse; if the spread persists, it is confirmed.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • RM evaluation should include downstream task metrics such as best-of-N accuracy and search-guiding score, not just pairwise or correctness accuracy.
  • Generative reward models, despite higher cost, are the safer choice when out-of-distribution generalization matters.
  • Process reward models are worth the extra step-level supervision for test-time selection, but should not be assumed to improve online RL over outcome rewards.
  • Improving the reasoning ability of the base model that serves as a generative RM should improve its judging accuracy, making reasoning training and reward-model training mutually reinforcing.
  • Most current RMs, especially discriminative ones, need task-specific retraining or domain adaptation when deployed outside their training distribution.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural extension: for online RL, a reward model's usefulness may hinge less on its ranking accuracy than on properties like reward variance and signal-to-noise ratio; the survey's cited evidence points this way but the authors do not make it their headline.
  • If correctness-style benchmarks continue to misalign with task performance, leaderboard rankings of RMs should probably be re-computed on downstream tasks across several policy models, and users should treat benchmark leader positions as weak evidence.
  • The co-evolution result suggests a concrete test: train the same base model alternately on generation and verification objectives and measure whether each stage raises the other; if it does, deliberate alternating training schedules could outperform separate reward-model and policy-model pipelines.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper is an analytical survey of reward models (RMs) for LLM reasoning. It develops taxonomies (discriminative vs. generative RMs, ORMs vs. PRMs, pointwise vs. pairwise), reviews evaluation benchmarks, and surveys three application areas: test-time guidance, synthetic data curation/self-improvement, and online RL. The paper also offers four analytical findings (Q1–Q4) on RM selection, OOD generalization, the co-evolution of generative and discriminative ability, and the adequacy of current RM evaluation metrics. The empirical contribution is small: Table V measures the correlation between generation and discrimination for ten LLMs, and Figure 5/Table IX compares ProcessBench correctness with downstream BoN/MCTS/Beam performance for six PRMs. The central empirical claim is that correctness-focused metrics, especially ProcessBench, may not predict real downstream performance, so practitioners should evaluate RMs with BoN-style metrics.

Significance. If the Q3 and Q4 findings hold, the survey provides actionable guidance for RM selection and evaluation, and it consolidates a rapidly growing literature in a useful way. The survey's taxonomies and coverage of recent methods/benchmarks are generally accurate and well organized. The paper's strengths include broad literature coverage, explicit comparisons of external results, and carefully hedged qualitative claims. However, the two new empirical analyses are small and lack statistical rigor, and the online-RL half of Q4 rests entirely on citations. The paper would be strengthened by making the new experiments reproducible and robust, or by explicitly scoping the 'we find' claims to the evidence provided. As a survey, the central claims remain defensible, but the novel empirical support needs attention.

major comments (3)
  1. [Section VI-D / Figure 5 / Table IX] The headline rank inversion — Skywork-PRM-7B 'ranks first in 4 out of 6 downstream tasks' despite a moderate ProcessBench score — is the key direct support for Q4, but Table IX reports no variance, no repeated seeds, and no significance tests. With six PRMs and margins as small as 1.0 pt (Qwen Beam: 82.2 vs 81.2; Mistral BoN: 49.6 vs 48.0), the inversion may be sampling noise. Please provide bootstrap confidence intervals, per-question error bars, or repeated-seed results, and state the exact number of MATH500 items used. If such robustness cannot be supplied, reframe this as a case study and lean on the cited external correlations.
  2. [Section VI-C / Table V] Q3's 'strong correlation' between generative and discriminative ability is asserted without a quantified correlation coefficient, confidence interval, or test. Discrimination is measured with a single fixed LLM-as-a-judge prompt that checks only final-answer correctness (Appendix C) on 100 random questions per dataset (Appendix B). This makes the trend vulnerable to prompt sensitivity and to contamination by the judge's own training data — an issue the paper itself raises for GPT-4o but does not resolve. Please report Spearman/pairwise correlations with uncertainty, vary judge prompts, and cross-check a subset with verifiable labels (e.g., exact-match). Also clarify the mixture of officially reported generation scores and 32-trial averages in the 'Avg.' column.
  3. [Section VI-D / Abstract / Q4] The claim that existing RM evaluation metrics are insufficient for online RL is supported only by external citations [235]–[237]; the authors' new experiments address test-time guidance (BoN/MCTS/Beam) exclusively. Since the abstract and introduction present Q4 as based partly on 'our empirical findings,' the paper should explicitly scope the claim: for test-time guidance the authors' own experiment is suggestive; for online RL the argument is a literature-based synthesis. This scoping is necessary to avoid overclaiming the paper's direct evidence.
minor comments (5)
  1. [Section II-C, Eq. (1)-(2)] Several formulas are incomplete or mis-rendered: `Rpoint_theta(p, tau) = r` and `Rpair_theta(P, tau1, tau2) = tau*` are stub equations, and Eq. (1) has an unmatched parenthesis in the expectation. Please fix the notation.
  2. [Appendix B] The paper does not state whether code, data, random seeds, or judge-prompt variants will be released. Given that Figure 5 and Table V are new empirical contributions, a reproducibility statement (even 'available upon request') is needed.
  3. [Tables III/IV] Values marked with * are described as read from published figures. This should be noted in the captions themselves, and the precision of such digitized values should be treated cautiously when drawing conclusions.
  4. [Appendix / Figure 6] Figure 6, the Spearman correlation heatmap among test-time strategies and ProcessBench, appears in the appendix but is not referenced in the main text. It would strengthen Section VI-D and should be cited there.
  5. [Section IV-A] Minor typo: 'REST-MCTS*' should be 'ReST-MCTS*' to match the referenced work. There are also occasional spacing/brace issues in the Appendix C prompts.

Circularity Check

0 steps flagged

No significant circularity: the survey's analytical claims rest on external citations and fresh experiments; any self-citations are list-level and non-load-bearing.

full rationale

The paper's derivation chain is not circular. The central empirical claim (Section VI-D) is that correctness metrics such as ProcessBench are insufficient to predict downstream test-time performance; this is supported by a new experiment (Figure 5, Table IX) that compares ProcessBench-MATH500 accuracy against BoN@8, MCTS, and beam-search accuracies on MATH500 for six PRMs. No fitted parameter is later renamed as a prediction: the linear-regression trend lines in Figure 5 are descriptive summaries, and the paper uses residual cases (e.g., Skywork-PRM-7B) to argue that correctness scores are insufficient, which is an ordinary empirical comparison rather than a constructional equivalence. The co-evolution claim in Section VI-C similarly rests on separately measured generation scores and LLM-as-a-judge discrimination accuracy (Table V, Appendix B); the correlation is empirical, not definitional. The only author-overlapping self-citations, notably SelfCheck [165] in the sampling/selection list and possibly RM-Bench [80] in Table I, appear in method enumerations and are not load-bearing: the main arguments about RM selection, generalization, evaluation, and online RL are supported by external references (e.g., [79], [81], [89], [34], [235], [236], [237]) and by the paper's own new experiments. Concerns such as the small number of PRMs in Table IX, the single MATH500 dataset, the absence of variance estimates, and the reliance on one LLM-as-a-judge prompt in Table V are threats to statistical robustness or measurement validity, but they are not circularity and do not indicate that the conclusions reduce to their inputs.

Axiom & Free-Parameter Ledger

4 free parameters · 3 axioms · 0 invented entities

No mathematical free parameters or invented entities. The empirical claims rest on several chosen experimental settings and representativeness assumptions, which are the main fragility points.

free parameters (4)
  • beam_size = 4
    Section VI-D/Appendix B: used for beam search; no sensitivity analysis provided; could affect PRM downstream rankings.
  • mcts_simulations = 4
    Section VI-D/Appendix B: MCTS used 4 simulation paths; no sensitivity analysis provided.
  • generation_temperature = 0.7
    Appendix B: set for all evaluations; may affect BoN and search results.
  • num_questions_per_dataset = 100
    Appendix B: 100 random questions per dataset for discrimination evaluation; small sample, no confidence intervals.
axioms (3)
  • domain assumption The cited literature accurately reports the results attributed to it.
    A survey's comparative claims (e.g., GRMs > DRMs, PRMs > ORMs at test time) rest on the reliability of the cited studies.
  • domain assumption The six PRMs and two policy models in Figure 5 are representative of the broader PRM landscape.
    Section VI-D: conclusions about ProcessBench vs downstream performance are based on this small set.
  • domain assumption The LLM-as-a-judge prompt in Appendix C measures discriminative ability rather than prompt-following or answer-format matching.
    Section VI-C: Table V correlations would change if the judge prompt rewarded step-level reasoning or penalized style artifacts.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Enhancing Large Language Model Reasoning with Reward Models: An Analytical Survey." pith.science (2026). https://pith.science/paper/U5QKEWGY

@misc{pith2026251001925,
  author       = {Pith},
  title        = {Pith review of: Enhancing Large Language Model Reasoning with Reward Models: An Analytical Survey},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/U5QKEWGY}},
  note         = {Machine review of arXiv:2510.01925}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Reward models (RMs) play a critical role in enhancing the reasoning performance of LLMs. For example, they can provide training signals to finetune LLMs during reinforcement learning (RL) and help select the best answer from multiple candidates during inference. In this paper, we provide a systematic introduction to RMs, along with a comprehensive survey of their applications in LLM reasoning. We first review fundamental concepts of RMs, including their architectures, training methodologies, and evaluation techniques. Then, we explore their key applications: (1) guiding generation and selecting optimal outputs during LLM inference, (2) facilitating data synthesis and iterative self-improvement for LLMs, and (3) providing training signals in RL-based finetuning. Finally, we discuss critical open questions regarding the selection, generalization, evaluation, and enhancement of RMs, based on existing research and our own empirical findings. Our analysis aims to provide actionable insights for the effective deployment and advancement of RMs for LLM reasoning.

Figures

Figures reproduced from arXiv: 2510.01925 by Hao Xu, Ning Miao, Qiyuan Liu, Wei Chen, Xuhong Chen, Yee Whye Teh.

Figure 1
Figure 1. Figure 1: Illustration of three main applications of reward models in LLM reasoning. Green/red blocks denote higher/lower-quality [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Taxonomy of current research on process reward models [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Applications of RMs in LLM reasoning frequently (i.e., a majority vote over final answers) without an explicit verifier. In contrast, the generator-verifier paradigm equips selection with reward scores from PRMs or ORMs to explicitly verify the correctness of each solution. Whereas self-consistency may fail when the policy model has a higher probability of generating incorrect answers, selection methods wi… view at source ↗
Figure 4
Figure 4. Figure 4: Comparisons of Llama and Qwen response styles in an example math question [PITH_FULL_IMAGE:figures/full_fig_p013_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: The relationship between correctness scores (ProcessBench), BoN scores, and search-guiding performance (MCTS [PITH_FULL_IMAGE:figures/full_fig_p015_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Heatmap illustrating Spearman correlation coefficients among the evaluated test-time search strategies and ProcessBench [PITH_FULL_IMAGE:figures/full_fig_p022_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: The discrimination accuracy for responses generated from different models [PITH_FULL_IMAGE:figures/full_fig_p023_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learning

    cs.LG 2026-04 unverdicted novelty 7.0

    This survey introduces the Generate-Filter-Control-Replay (GFCR) taxonomy to structure rollout pipelines for RL-based post-training of reasoning LLMs.

  2. Process Advantage Signal Shaping: A Paradigm-Agnostic Middleware for Process-Supervised RL in LLM Reasoners

    cs.AI 2026-06 unverdicted novelty 6.0

    PASS middleware independently standardizes process/outcome/format streams, derives value-homogeneous chunks, and converts cumulative returns to average value density, yielding consistent pass@1 gains over GRPO baselin...

Reference graph

Works this paper leans on

237 extracted references · cited by 2 Pith papers

  1. [1]

    Sparks of artificial general intelligence: Early experiments with gpt-4,

    S. Bubeck, V . Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y . T. Lee, Y . Li, S. Lundberg, H. Nori, H. Palangi, M. T. Ribeiro, and Y . Zhang, “Sparks of artificial general intelligence: Early experiments with gpt-4,” 2023

    2023

  2. [2]

    A survey on medical large language models: Technology, application, trustworthiness, and future directions,

    L. Liu, X. Yang, J. Lei, Y . Shen, J. Wang, P. Wei, Z. Chu, Z. Qin, and K. Ren, “A survey on medical large language models: Technology, application, trustworthiness, and future directions,” 2024

    2024

  3. [3]

    Vision-language models for vision tasks: A survey,

    J. Zhang, J. Huang, S. Jin, and S. Lu, “Vision-language models for vision tasks: A survey,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, no. 8, pp. 5625–5644, 2024

    2024

  4. [4]

    Bridging the linguistic divide: A survey on leveraging large language models for machine translation,

    B. Gain, D. Bandyopadhyay, and A. Ekbal, “Bridging the linguistic divide: A survey on leveraging large language models for machine translation,” 2025

    2025

  5. [5]

    A survey of large language model agents for question answering,

    M. Yue, “A survey of large language model agents for question answering,” 2025

    2025

  6. [6]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou, “Chain-of-thought prompting elicits reasoning in large language models,” 2023. 16

    2023

  7. [7]

    Tree of thoughts: Deliberate problem solving with large language models,

    S. Yao, D. Yu, J. Zhao, I. Shafran, T. L. Griffiths, Y . Cao, and K. R. Narasimhan, “Tree of thoughts: Deliberate problem solving with large language models,” inThirty-seventh Conference on Neural Information Processing Systems, 2023

    2023

  8. [8]

    Solving quantitative reasoning problems with language models,

    A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V . Ramasesh, A. Slone, C. Anil, I. Schlag, T. Gutman-Solo, Y . Wu, B. Neyshabur, G. Gur-Ari, and V . Misra, “Solving quantitative reasoning problems with language models,” 2022

    2022

  9. [9]

    Mint: Boosting generalization in mathematical reasoning via multi-view fine- tuning,

    Z. Liang, D. Yu, X. Pan, W. Yao, Q. Zeng, X. Zhang, and D. Yu, “Mint: Boosting generalization in mathematical reasoning via multi-view fine- tuning,” 2023

    2023

  10. [10]

    Robust visual question answering: Datasets, methods, and future challenges,

    J. Ma, P. Wang, D. Kong, Z. Wang, J. Liu, H. Pei, and J. Zhao, “Robust visual question answering: Datasets, methods, and future challenges,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, no. 8, pp. 5575–5594, 2024

    2024

  11. [11]

    Learning from mistakes makes llm better reasoner,

    S. An, Z. Ma, Z. Lin, N. Zheng, J.-G. Lou, and W. Chen, “Learning from mistakes makes llm better reasoner,” 2024

    2024

  12. [12]

    Openai o1 system card,

    OpenAIet al., “Openai o1 system card,” 2024

    2024

  13. [13]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,

    DeepSeek-AIet al., “Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,” 2025

    2025

  14. [14]

    Tulu 3: Pushing frontiers in open language model post-training,

    N. Lambert, J. Morrison, V . Pyatkin, S. Huang, H. Ivison, F. Brahman, L. J. V . Miranda, A. Liu, N. Dziri, S. Lyu, Y . Gu, S. Malik, V . Graf, J. D. Hwang, J. Yang, R. L. Bras, O. Tafjord, C. Wilhelm, L. Soldaini, N. A. Smith, Y . Wang, P. Dasigi, and H. Hajishirzi, “Tulu 3: Pushing frontiers in open language model post-training,” 2025

    2025

  15. [15]

    Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback,

    H. Lee, S. Phatale, H. Mansoor, T. Mesnard, J. Ferret, K. Lu, C. Bishop, E. Hall, V . Carbune, A. Rastogi, and S. Prakash, “Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback,” 2024

    2024

  16. [16]

    Improve mathematical reasoning in language models by automated process supervision,

    L. Luo, Y . Liu, R. Liu, S. Phatale, M. Guo, H. Lara, Y . Li, L. Shu, Y . Zhu, L. Meng, J. Sun, and A. Rastogi, “Improve mathematical reasoning in language models by automated process supervision,” 2024

    2024

  17. [17]

    Advancing process verification for large language models via tree-based preference learning,

    M. He, Y . Shen, W. Zhang, Z. Tan, and W. Lu, “Advancing process verification for large language models via tree-based preference learning,” inProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y . Al-Onaizan, M. Bansal, and Y .-N. Chen, Eds. Miami, Florida, USA: Association for Computational Linguistics, Nov. 2024, pp...

    2024

  18. [18]

    Token-supervised value models for enhancing mathematical problem- solving capabilities of large language models,

    J. H. Lee, J. Y . Yang, B. Heo, D. Han, K. Kim, E. Yang, and K. M. Yoo, “Token-supervised value models for enhancing mathematical problem- solving capabilities of large language models,” 2025

    2025

  19. [19]

    Coarse-to-fine process reward modeling for mathematical reasoning,

    Y . Hu, G. Chen, J. Zhao, S. Ouyang, and Y . Liu, “Coarse-to-fine process reward modeling for mathematical reasoning,” 2025

    2025

  20. [20]

    Visualprm: An effective process reward model for multimodal reasoning,

    W. Wang, Z. Gao, L. Chen, Z. Chen, J. Zhu, X. Zhao, Y . Liu, Y . Cao, S. Ye, X. Zhu, L. Lu, H. Duan, Y . Qiao, J. Dai, and W. Wang, “Visualprm: An effective process reward model for multimodal reasoning,” 2025

    2025

  21. [21]

    Towards hierarchical multi-step reward models for enhanced reasoning in large language models,

    T. Wang, Z. Jiang, Z. He, W. Yang, Y . Zheng, Z. Li, Z. He, S. Tong, and H. Gong, “Towards hierarchical multi-step reward models for enhanced reasoning in large language models,” 2025

    2025

  22. [22]

    Adaptivestep: Automatically dividing reasoning step through model confidence,

    Y . Liu, J. Lu, Z. Chen, C. Qu, J. K. Liu, C. Liu, Z. Cai, Y . Xia, L. Zhao, J. Bian, C. Zhang, W. Shen, and Z. Lin, “Adaptivestep: Automatically dividing reasoning step through model confidence,” 2025

    2025

  23. [23]

    Vilbench: A suite for vision-language process reward modeling,

    H. Tu, W. Feng, H. Chen, H. Liu, X. Tang, and C. Xie, “Vilbench: A suite for vision-language process reward modeling,” 2025

    2025

  24. [24]

    Retrieval-augmented process reward model for generalizable mathematical reasoning,

    J. Zhu, C. Zheng, J. Lin, K. Du, Y . Wen, Y . Yu, J. Wang, and W. Zhang, “Retrieval-augmented process reward model for generalizable mathematical reasoning,” 2025

    2025

  25. [25]

    Making large language models better reasoners with step-aware verifier,

    Y . Li, Z. Lin, S. Zhang, Q. Fu, B. Chen, J.-G. Lou, and W. Chen, “Making large language models better reasoners with step-aware verifier,” 2023

    2023

  26. [26]

    OVM, outcome-supervised value models for planning in mathematical reasoning,

    F. Yu, A. Gao, and B. Wang, “OVM, outcome-supervised value models for planning in mathematical reasoning,” inFindings of the Association for Computational Linguistics: NAACL 2024, K. Duh, H. Gomez, and S. Bethard, Eds. Mexico City, Mexico: Association for Computational Linguistics, Jun. 2024, pp. 858–875

    2024

  27. [27]

    Let’s verify step by step,

    H. Lightman, V . Kosaraju, Y . Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe, “Let’s verify step by step,” 2023

    2023

  28. [28]

    Math-shepherd: Verify and reinforce LLMs step-by-step without human annotations,

    P. Wang, L. Li, Z. Shao, R. Xu, D. Dai, Y . Li, D. Chen, Y . Wu, and Z. Sui, “Math-shepherd: Verify and reinforce LLMs step-by-step without human annotations,” inProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), L.-W. Ku, A. Martins, and V . Srikumar, Eds. Bangkok, Thailand: Association for Co...

    2024

  29. [29]

    Multi-step problem solving through a verifier: An empirical analysis on model- induced process supervision,

    Z. Wang, Y . Li, Y . Wu, L. Luo, L. Hou, H. Yu, and J. Shang, “Multi-step problem solving through a verifier: An empirical analysis on model- induced process supervision,” 2024

    2024

  30. [30]

    Glore: When, where, and how to improve llm reasoning via global and local refinements,

    A. Havrilla, S. Raparthy, C. Nalmpantis, J. Dwivedi-Yu, M. Zhuravin- skyi, E. Hambro, and R. Raileanu, “Glore: When, where, and how to improve llm reasoning via global and local refinements,” 2024

    2024

  31. [31]

    Autopsv: Automated process-supervised verifier,

    J. Lu, Z. Dou, H. Wang, Z. Cao, J. Dai, Y . Wan, Y . Feng, and Z. Guo, “Autopsv: Automated process-supervised verifier,” inAdvances in Neural Information Processing Systems, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, Eds., vol. 37. Curran Associates, Inc., 2024, pp. 79 935–79 962

    2024

  32. [32]

    Rewarding progress: Scaling automated process verifiers for llm reasoning,

    A. Setlur, C. Nagpal, A. Fisch, X. Geng, J. Eisenstein, R. Agarwal, A. Agarwal, J. Berant, and A. Kumar, “Rewarding progress: Scaling automated process verifiers for llm reasoning,” 2024

    2024

  33. [33]

    Entropy-regularized process reward model,

    H. Zhang, P. Wang, S. Diao, Y . Lin, R. Pan, H. Dong, D. Zhang, P. Molchanov, and T. Zhang, “Entropy-regularized process reward model,” 2024

    2024

  34. [34]

    The lessons of developing process reward models in mathematical reasoning,

    Z. Zhang, C. Zheng, Y . Wu, B. Zhang, R. Lin, B. Yu, D. Liu, J. Zhou, and J. Lin, “The lessons of developing process reward models in mathematical reasoning,” 2025

    2025

  35. [35]

    Athena: Enhancing multimodal reasoning with data-efficient process reward models,

    S. Wang, Z. Liu, J. Wei, X. Yin, D. Li, and E. Barsoum, “Athena: Enhancing multimodal reasoning with data-efficient process reward models,” 2025

    2025

  36. [36]

    Reasonflux- prm: Trajectory-aware prms for long chain-of-thought reasoning in llms,

    J. Zou, L. Yang, J. Gu, J. Qiu, K. Shen, J. He, and M. Wang, “Reasonflux- prm: Trajectory-aware prms for long chain-of-thought reasoning in llms,” 2025

    2025

  37. [37]

    Better process supervision with bi-directional rewarding signals,

    W. Chen, W. He, Z. Xi, H. Guo, B. Hong, J. Zhang, R. Zheng, N. Li, T. Gui, Y . Li, Q. Zhang, and X. Huang, “Better process supervision with bi-directional rewarding signals,” 2025

    2025

  38. [38]

    Duashepherd: Integrating stepwise correctness and potential rewards for mathematical reasoning,

    Y . Wu, J. Song, H. Zhang, T. Zhang, and C. Niu, “Duashepherd: Integrating stepwise correctness and potential rewards for mathematical reasoning,” 2025

    2025

  39. [39]

    Llm critics help catch bugs in mathematics: Towards a better mathematical verifier with natural language feedback,

    B. Gao, Z. Cai, R. Xu, P. Wang, C. Zheng, R. Lin, K. Lu, D. Liu, C. Zhou, W. Xiao, J. Hu, T. Liu, and B. Chang, “Llm critics help catch bugs in mathematics: Towards a better mathematical verifier with natural language feedback,” 2024

    2024

  40. [40]

    Verifierq: Enhancing llm test time compute with q-learning-based verifiers,

    J. Qi, H. Tang, and Z. Zhu, “Verifierq: Enhancing llm test time compute with q-learning-based verifiers,” 2024

    2024

  41. [41]

    Process reward model with q-value rankings,

    W. Li and Y . Li, “Process reward model with q-value rankings,” 2025

    2025

  42. [42]

    Free process rewards without process labels,

    L. Yuan, W. Li, H. Chen, G. Cui, N. Ding, K. Zhang, B. Zhou, Z. Liu, and H. Peng, “Free process rewards without process labels,” 2024

    2024

  43. [43]

    Tdrm: Smooth reward models with temporal difference for llm rl and inference,

    D. Zhang, M. Cai, J. Li, Z. Hu, Y . Yue, Y . Dong, and J. Tang, “Tdrm: Smooth reward models with temporal difference for llm rl and inference,” 2025

    2025

  44. [44]

    Cold: Counterfactually-guided length debiasing for process reward models,

    C. Zheng, J. Zhu, J. Lin, X. Dai, Y . Yu, W. Zhang, and M. Yang, “Cold: Counterfactually-guided length debiasing for process reward models,” 2025

    2025

  45. [45]

    Judging LLM-as-a-judge with MT-bench and chatbot arena,

    L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica, “Judging LLM-as-a-judge with MT-bench and chatbot arena,” inThirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023

    2023

  46. [46]

    R-prm: Reasoning-driven process reward modeling,

    S. She, J. Liu, Y . Liu, J. Chen, X. Huang, and S. Huang, “R-prm: Reasoning-driven process reward modeling,” 2025

    2025

  47. [47]

    Genprm: Scaling test-time compute of process reward models via generative reasoning,

    J. Zhao, R. Liu, K. Zhang, Z. Zhou, J. Gao, D. Li, J. Lyu, Z. Qian, B. Qi, X. Li, and B. Zhou, “Genprm: Scaling test-time compute of process reward models via generative reasoning,” 2025

    2025

  48. [48]

    Scaling evaluation-time compute with reasoning models as process evaluators,

    S. Kim, I. Wu, J. Lee, X. Yue, S. Lee, M. Moon, K. Gashteovski, C. Lawrence, J. Hockenmaier, G. Neubig, and S. Welleck, “Scaling evaluation-time compute with reasoning models as process evaluators,” 2025

    2025

  49. [49]

    Spc: Evolving self-play critic via adversarial games for llm reasoning,

    J. Chen, B. Zhang, R. Ma, P. Wang, X. Liang, Z. Tu, X. Li, and K.-Y . K. Wong, “Spc: Evolving self-play critic via adversarial games for llm reasoning,” 2025

    2025

  50. [50]

    Process reward models that think,

    M. Khalifa, R. Agarwal, L. Logeswaran, J. Kim, H. Peng, M. Lee, H. Lee, and L. Wang, “Process reward models that think,” 2025

    2025

  51. [51]

    Stepwiser: Stepwise generative judges for wiser reasoning,

    W. Xiong, W. Zhao, W. Yuan, O. Golovneva, T. Zhang, J. Weston, and S. Sukhbaatar, “Stepwiser: Stepwise generative judges for wiser reasoning,” 2025

    2025

  52. [52]

    Solving math word problems with process- and outcome-based feedback,

    J. Uesato, N. Kushman, R. Kumar, F. Song, N. Siegel, L. Wang, A. Creswell, G. Irving, and I. Higgins, “Solving math word problems with process- and outcome-based feedback,” 2022

    2022

  53. [53]

    Training verifiers to solve math word problems,

    K. Cobbe, V . Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman, “Training verifiers to solve math word problems,” 2021

    2021

  54. [54]

    Direct preference optimization: Your language model is secretly a reward model,

    R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn, “Direct preference optimization: Your language model is secretly a reward model,” 2024

    2024

  55. [55]

    Inference-time scaling for generalist reward modeling,

    Z. Liu, P. Wang, R. Xu, S. Ma, C. Ruan, P. Li, Y . Liu, and Y . Wu, “Inference-time scaling for generalist reward modeling,” 2025. 17

    2025

  56. [56]

    Rm-r1: Reward modeling as reasoning,

    X. Chen, G. Li, Z. Wang, B. Jin, C. Qian, Y . Wang, H. Wang, Y . Zhang, D. Zhang, T. Zhang, H. Tong, and H. Ji, “Rm-r1: Reward modeling as reasoning,” 2025

    2025

  57. [57]

    Ticking all the boxes: Generated checklists improve llm evaluation and generation,

    J. Cook, T. Rocktäschel, J. Foerster, D. Aumiller, and A. Wang, “Ticking all the boxes: Generated checklists improve llm evaluation and generation,” 2024

    2024

  58. [58]

    Generative verifiers: Reward modeling as next-token prediction,

    L. Zhang, A. Hosseini, H. Bansal, M. Kazemi, A. Kumar, and R. Agarwal, “Generative verifiers: Reward modeling as next-token prediction,” 2025

    2025

  59. [59]

    Critique- out-loud reward models,

    Z. Ankner, M. Paul, B. Cui, J. D. Chang, and P. Ammanabrolu, “Critique- out-loud reward models,” 2024

    2024

  60. [60]

    Learning to reason for factuality,

    X. Chen, I. Kulikov, V .-P. Berges, B. O ˘guz, R. Shao, G. Ghosh, J. Weston, and W. tau Yih, “Learning to reason for factuality,” 2025

    2025

  61. [61]

    Internlm2 technical report,

    Z. Caiet al., “Internlm2 technical report,” 2024

    2024

  62. [62]

    Advancing llm reasoning generalists with preference trees,

    L. Yuan, G. Cui, H. Wang, N. Ding, X. Wang, J. Deng, B. Shan, H. Chen, R. Xie, Y . Lin, Z. Liu, B. Zhou, H. Peng, Z. Liu, and M. Sun, “Advancing llm reasoning generalists with preference trees,” 2024

    2024

  63. [63]

    Interpretable preferences via multi-objective reward modeling and mixture-of-experts,

    H. Wang, W. Xiong, T. Xie, H. Zhao, and T. Zhang, “Interpretable preferences via multi-objective reward modeling and mixture-of-experts,” 2024

    2024

  64. [64]

    Llm-blender: Ensembling large language models with pairwise ranking and generative fusion,

    D. Jiang, X. Ren, and B. Y . Lin, “Llm-blender: Ensembling large language models with pairwise ranking and generative fusion,” 2023

    2023

  65. [65]

    Helpsteer2-preference: Complementing ratings with preferences,

    Z. Wang, A. Bukharin, O. Delalleau, D. Egert, G. Shen, J. Zeng, O. Kuchaiev, and Y . Dong, “Helpsteer2-preference: Complementing ratings with preferences,” 2025

    2025

  66. [66]

    Kto: Model alignment as prospect theoretic optimization,

    K. Ethayarajh, W. Xu, N. Muennighoff, D. Jurafsky, and D. Kiela, “Kto: Model alignment as prospect theoretic optimization,” 2024

    2024

  67. [67]

    Bootstrapping language models with dpo implicit rewards,

    C. Chen, Z. Liu, C. Du, T. Pang, Q. Liu, A. Sinha, P. Varakantham, and M. Lin, “Bootstrapping language models with dpo implicit rewards,” 2025

    2025

  68. [68]

    Generative judge for evaluating alignment,

    J. Li, S. Sun, W. Yuan, R.-Z. Fan, hai zhao, and P. Liu, “Generative judge for evaluating alignment,” inThe Twelfth International Conference on Learning Representations, 2024

    2024

  69. [69]

    Prometheus 2: An open source language model specialized in evaluating other language models,

    S. Kim, J. Suk, S. Longpre, B. Y . Lin, J. Shin, S. Welleck, G. Neubig, M. Lee, K. Lee, and M. Seo, “Prometheus 2: An open source language model specialized in evaluating other language models,” inProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y . Al-Onaizan, M. Bansal, and Y .-N. Chen, Eds. Miami, Florida, USA: As...

    2024

  70. [70]

    Foundational autoraters: Taming large language models for better automatic evaluation,

    T. Vu, K. Krishna, S. Alzubi, C. Tar, M. Faruqui, and Y .-H. Sung, “Foundational autoraters: Taming large language models for better automatic evaluation,” inProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y . Al-Onaizan, M. Bansal, and Y .-N. Chen, Eds. Miami, Florida, USA: Association for Computational Linguistics...

    2024

  71. [71]

    Compassjudger-1: All-in-one judge model helps model evaluation and evolution,

    M. Cao, A. Lam, H. Duan, H. Liu, S. Zhang, and K. Chen, “Compassjudger-1: All-in-one judge model helps model evaluation and evolution,” 2024

    2024

  72. [72]

    Learning LLM-as-a-judge for preference alignment,

    Z. Ye, X. Li, Q. Li, Q. Ai, Y . Zhou, W. Shen, D. Yan, and Y . LIU, “Learning LLM-as-a-judge for preference alignment,” inThe Thirteenth International Conference on Learning Representations, 2025

    2025

  73. [73]

    Atla selene mini: A general purpose evaluation model,

    A. Alexandru, A. Calvi, H. Broomfield, J. Golden, K. Dai, M. Leys, M. Burger, M. Bartolo, R. Engeler, S. Pisupati, T. Drane, and Y . S. Park, “Atla selene mini: A general purpose evaluation model,” 2025

    2025

  74. [74]

    One token to fool llm-as-a-judge,

    Y . Zhao, H. Liu, D. Yu, S. Y . Kung, H. Mi, and D. Yu, “One token to fool llm-as-a-judge,” 2025

    2025

  75. [75]

    Judgelrm: Large reasoning models as a judge,

    N. Chen, Z. Hu, Q. Zou, J. Wu, Q. Wang, B. Hooi, and B. He, “Judgelrm: Large reasoning models as a judge,” 2025

    2025

  76. [76]

    Unified multimodal chain-of-thought reward model through reinforcement fine- tuning,

    Y . Wang, Z. Li, Y . Zang, C. Wang, Q. Lu, C. Jin, and J. Wang, “Unified multimodal chain-of-thought reward model through reinforcement fine- tuning,” 2025

    2025

  77. [78]

    Pairjudge rm: Perform best-of-n sampling with knockout tournament,

    Y . Liu, Z. Yao, R. Min, Y . Cao, L. Hou, and J. Li, “Pairjudge rm: Perform best-of-n sampling with knockout tournament,” 2025

    2025

  78. [79]

    Rewardbench: Evaluating reward models for language modeling,

    N. Lambert, V . Pyatkin, J. Morrison, L. Miranda, B. Y . Lin, K. Chandu, N. Dziri, S. Kumar, T. Zick, Y . Choi, N. A. Smith, and H. Hajishirzi, “Rewardbench: Evaluating reward models for language modeling,” 2024

    2024

  79. [80]

    Rm-bench: Benchmarking reward models of language models with subtlety and style,

    Y . Liu, Z. Yao, R. Min, Y . Cao, L. Hou, and J. Li, “Rm-bench: Benchmarking reward models of language models with subtlety and style,” 2024

    2024

  80. [81]

    Rmb: Comprehensively benchmarking reward models in llm alignment,

    E. Zhou, G. Zheng, B. Wang, Z. Xi, S. Dou, R. Bao, W. Shen, L. Xiong, J. Fan, Y . Mou, R. Zheng, T. Gui, Q. Zhang, and X. Huang, “Rmb: Comprehensively benchmarking reward models in llm alignment,” 2025

    2025

Showing first 80 references.

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.